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Introduction 

This  project  is  broken  into  three  focus  areas:  robotic  curriculum,  telesurgery,  and  simulation.  In 
each  we  are  exploring  various  applications  and  extensions  of  the  existing  robotic  surgical 
systems.  Under  robotic  curriculum  we  are  bringing  together  the  leading  surgeons  and 
academicians  to  define  the  outcomes  measures,  curriculum,  psychomotor  devices,  and  high 
stakes  testing  that  should  be  used  to  certify  surgeons  who  wish  to  practice  robotic  surgery.  Under 
telesurgery  we  are  exploring  the  ability  to  perform  telesurgery  using  a  robot  within  a 
metropolitan  area  based  on  the  currently  available  technology.  Under  simulation  we  are 
examining  the  impact  of  rehearsing  a  procedure  in  a  simulator  immediately  before  performing 
that  same  procedure  on  a  patient.  This  area  also  includes  a  comparative  evaluation  of  all  of  the 
robotic  simulators  that  are  available  with  a  recommendation  of  the  best  fit  for  military  surgeons. 
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Statement  of  Work 


ORIGINAL  STATEMENT  OF  WORK 

There  are  three  primary  areas  of  this  research:  Telesurgery,  Simulation,  and  Robotic  Curriculum. 
(1)  The  telesurgery  project  will  identify  the  characteristics  of  latency  during  telesurgery  and 
investigate  the  application  of  principles  of  automatic  surgery.  (2)  Under  simulation,  we  will 
validate  a  simulator  that  can  be  used  by  military  surgeons  to  maintain  their  robotic  skills  while 
deployed.  We  will  then  use  this  device  to  explore  the  feasibility  of  surgical  rehearsal  as  a 
potential  solution  to  the  latency  issue  in  telesurgery.  (3)  We  will  organize  robotic  surgery 
experts  to  develop  a  nationally  accepted  curriculum  in  the  Fundamentals  of  Robotic  Surgery 
(FRS). 

Period  1 

Telesurgery:  Communications  Latency  Experiments.  Identify  communication  latency,  measure 
safe  latency  levels  for  each  robotic  movement,  modify  surgical  procedures  to  be  effective  in  this 
environment. 

Milestone:  Telesurgery  latency  experiment  report.  Award  +  270  days 

Simulation:  Military -use  Validation.  Validate  a  robotic  simulator  for  maintaining  the  robotic 
surgery  skills  of  deployed  military  surgeons. 

Milestone:  Robotic  simulator  validation  report.  Award  +  210  days 

Robotic  Curriculum:  Consensus  Conferences.  Organize  and  host  conferences  of  approximately 
40  leading  robotic  surgeons  from  around  the  United  States  to  include  military  surgeons.  Identify 
the  fundamental  knowledge  and  skills  that  should  be  a  foundation  for  every  robotic  surgeon. 
Milestone:  FRS  consensus  conference  reports.  Award  +180  days  and  365  days 

Period  2 

Telesurgery:  Automatic  Surgery.  Apply  movements  recorded  in  a  robotic  simulator  to  actual 
execution  with  the  da  Vinci  robot  on  solid  models.  Explore  ability  to  automatically  execute 
surgery  from  a  simulator  recording. 

Milestone:  Automatic  surgery  experiment  results.  Award  +  730  days 

Simulation:  Surgical  Rehearsal.  Experiment  with  the  effectiveness  of  simulated  surgical 
rehearsal  on  improving  the  outcomes  of  robotic  surgery. 

Milestone:  Surgical  rehearsal  experiment  results.  Award  +  540  days 

FRS  Curriculum  Validation  and  Transition.  Develop  specific  training  tasks  and  passing  criteria 
for  the  FRS  curriculum.  Process  the  curriculum  through  the  certifying  bodies. 

Milestone:  Telesurgery  medical  procedure  results.  Award  +  730  days 
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Telesurgery 

Communications  Latency  Experiments. 


All  telesurgery  work  was  completed  and  reported  in  the  previous  year  ending  August  2013.  See 
the  previous  annual  report  for  details  on  this  project. 


During  this  current  year  the  results  of  that  work  have  been  organized  into  a  new  conference 
presentation  and  a  new  journal  submission.  Those  items  are  included  in  the  appendix  of 
published  works  in  this  report. 

Automatic  Surgery. 

[Repeated  from  Sept  2013  annual  report] 

In  the  early  phases  of  this  project,  the  government  COR  encouraged  us  to  reconsider  our  efforts 
to  explore  automatic  surgery.  They  felt  that  the  robotic  and  simulator  technologies  currently 
available  clearly  indicated  that  experiments  in  this  area  would  be  premature  because  the  outcome 
was  expected  to  be  negative.  The  government  felt  that  the  knowledge  gained  would  not  justify 
the  funds  expended.  Our  initial  investigations  into  designing  this  experiment,  including  extensive 
discussions  with  the  manufactures  of  the  simulator  and  the  robot,  convinced  us  that  the 
government  was  correct  in  this  assessment.  As  a  result,  we  did  not  perform  this  experiment.  The 
funds  originally  scheduled  for  this  experiment  were  reallocated  to  other  experiments. 
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Simulation:  Military-use  Validation 

We  are  conducting  a  three  part  comparative  evaluation  of  the  available  robotic  simulator  devices. 
The  first  part  of  this  study  was  an  evaluation  of  the  system  capabilities  of  the  devices.  This  work 
was  delivered  as  a  report  to  USA  TATRC  in  August  2013.  The  complete  report  was  included  in 
the  Sept  2013  annual  report. 

The  second  part  was  a  subjective  evaluation  of  all  three  of  the  simulators  by  MD’s.  The  results  of 
that  comparison  have  been  prepared  in  a  report  and  presented  at  medical  conferences.  In 
summary,  as  a  mode  of  teaching  robotic  surgery,  the  participants  preferred  the  da  Vinci 
Simulation  System  (DVSS)  which  attaches  to  the  real  surgical  console.  They  found  the  dV- 
Trainer  simulator  to  be  acceptable  for  teaching  and  felt  that  it  provided  a  significant  economic 
advantage  since  it  is  stand  alone  and  does  not  require  access  to  a  da  Vinci  surgeon’s  console. 
They  felt  that  the  RoSS  simulator  was  a  poor  device  which  did  not  effectively  replicate  the 
performance  of  the  real  da  Vinci  robot  and  could  not  be  effectively  used  in  training. 

We  have  captured  the  results  of  this  experiment  in  a  paper  to  be  presented  at  the  2014  I/ITSEC 
Conference  (Orlando,  FL).  The  tables  and  graph  below  summarize  the  major  findings.  For  a 
complete  discussion  see  the  copy  of  the  paper  which  is  included  in  the  appendix  of  this  report. 


Table  2.  Average  scores  from  a  5-point  Likert  scale  on  face  validity. 


DVSS 

dV-Trainer 

RoSS 

Ql:  The  hand  controllers  on  this  simulator  are  effective  for 
working  in  the  simulated  environment. 

4.80 

3.62 

2.17 

Q4:  The  device  is  a  sufficiently  accurate  representation  of 
the  real  robotic  system. 

4.65 

3.45 

1.82 

Table  3.  Scores  on  a  5  point  Likert  scale  for  content  validity  questions. 


Likert  Score 

Strong  Dis 

Disagree 

Neither 

Agree 

Strong  Agree 

Q2:  The  3D  graphical  exercises  in  the  simulator  are  effective  for  teaching  robotic  skills. 

DVSS 

0% 

0% 

0% 

35.3% 

64.7% 

dV-Trainer 

2.9% 

5.9% 

11.8% 

50.0% 

29.4% 

RoSS 

20.6% 

38.2% 

17.6% 

17.6% 

5.9% 

Q5:  The  scoring  system  effectively  communicates  my  performance  on  the  exercise. 

DVSS 

2.9% 

5.9% 

2.9% 

38.2% 

50.0% 

dV-Trainer 

2.9% 

2.9% 

14.7% 

55.9% 

23.5% 

RoSS 

17.6% 

20.6% 

26.5% 

29.4% 

5.9% 

Q6:  The  scoring  system  effectively  guides  me  to  improve  performance  on  the  simulator. 

DVSS 

0% 

0% 

8.8% 

61.8% 

29.4% 

dV-Trainer 

2.9% 

2.9% 

11.8% 

61.8% 

20.6% 

RoSS 

18.2% 

18.2% 

27.3% 

33.3% 

3.0% 

Table  4.  Mann- Whitney  U  test  level  of  significance  on  construct  validity  measures 


DVSS 

dV-Trainer 

RoSS 

Time  to  Complete 

pO.001 

p<0.001 

p=0.221 
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Overall  Score 

p<0.01 

p=0.061 

n/a 

Economy  of  Motion 

p=0.216 

p<0.001 

p=0.566 

Number  of  Errors 

n/a 

n/a 

p=0.644 

Table  5.  Correlation  between  level  of  experience  and  simulator  scores 


DVSS 

dV-Trainer 

RoSS 

Overall  Score 

p=0.001 

p=0.031 

n/a 

Time  to  Complete 

p<0.001 

p<0.001 

p=0.181 

Economy  of  Motion 

II 

p 

o 

f/1 

p<0.001 

p=0.390 

Number  of  Errors 

n/a 

n/a 

p=0.563 

The  third  part  of  this  study  is  an  objective  evaluation  of  the  effectiveness  of  each  simulator  to 
improve  the  skills  of  a  robotic  surgeon.  In  this  part  we  are  measuring  the  amount  of  time  and 
effort  required  for  a  surgeon  to  reach  competency  levels  in  the  simulator.  The  subjects  will  return 
to  the  experiment  every  two  weeks  to  measure  the  degree  to  which  skills  are  retained  and  the 
amount  of  retraining  that  is  necessary  to  re-attain  competency.  This  information  will  assist  in  the 
development  of  a  training  protocol  to  maintain  robotic  competency,  such  as  during  the 
deployment  of  a  robotic  surgeon.  This  experiment  is  currently  underway. 
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Simulation:  Surgical  Rehearsal 

This  experiment  compared  the  effectiveness  of  traditional  classroom  training  to  simulation-based 
training  in  preparing  for  a  specific  procedure.  We  designed  the  experiment  to  be  carried  out  in 
conjunction  with  existing  educational  events  using  animals.  This  allowed  us  to  perform  the 
experiment  without  sacrificing  any  additional  animals  in  the  conduct  of  this  study. 

This  study  completed  with  125  subjects  (control=64,  experimental  1)  participating  in  the 
control  and  experimental  groups.  We  found  no  statistical  difference  in  the  performance  of  those 
who  prepared  via  traditional  classroom  methods  and  those  who  prepared  via  simulation-based 
rehearsal. 


Surgical  Rehearsal 


Didactic 


Simulation 


This  experiment  required  that  we  identify  a  small  procedure  which  existed  as  a  simulation 
exercise  and  could  be  performed  in  an  animal  model.  The  most  procedure-specific  exercise  in  the 
dV-Trainer  simulator  is  the  suturing  of  an  incision.  This  drove  the  selection  of  the  procedure 
which  would  be  used  to  measure  performance  on  this  experiment.  We  believe  that  the 
experiment  showed  no  difference  between  the  two  groups  because  the  suturing  procedure  was 
too  easy  for  the  experienced  subjects  who  were  enrolled.  However,  due  to  the  limited  availability 
of  simulated  procedures,  it  is  not  possible  to  create  a  more  rigorous  task  to  measure  at  this  time. 
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Robotic  Curriculum 

We  have  completed  the  development  of  an  online  robotic  surgery  curriculum  and  a  psychomotor 
skills  testing  device.  Both  of  these  were  created  through  collaboration  with  leading  robotic 
surgeons  from  around  the  world.  These  products  have  been  transferred  to  surgical  professional 
societies  who  are  organizing  a  multi-site  validation  trail  for  the  materials.  This  DoD  grant  is  not 
providing  funding  to  those  validation  trials.  They  are  being  carried  out  with  commercial  grant 
funding. 

Online  Curriculum 

The  online  curriculum  is  open  and  available  to  all  interested  parties.  It  can  be  accessed  at: 
http://FRSurgery.com/ 


Modules  of  the  FRS  Curriculum 


Module  1:  Introduction  to  Robotic  Surgical  Systems 


Module  3:  Psychomotor  Skills  Curriculum  Module  4:  Team  Training  and  Communication  Skills 

®FRS 


Psychomotor  Skills  Device 

The  psychomotor  device  has  been  completed  and  is  in  limited  production.  The  development  of 
this  device  has  conducted  under  the  funding  of  this  grant.  But  government  funds  are  not  being 
used  to  produce  these  devices  for  use  in  the  validation  trials.  Currently  all  production  capacity  is 
being  used  to  fill  the  order  of  the  sites  which  are  conducting  the  validation  trials  (not  part  of  this 
government  funded  grant). 

As  production  capacities  increase  we  will  make  the  device  available  for  commercial  purchase  by 

all  interested  parties.  Orders  can  be  placed  at: 

https://www.nicholsoncenter.com/ffskit 
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Simulator  Exercises 

During  the  development  of  the  physical  psychomotor  device,  two  simulator  companies  showed 
an  interest  in  adding  the  device  to  their  portfolio  of  virtual  exercises.  As  a  result,  using  their  own 
funding,  these  companies  have  created  virtual  exercises  that  match  the  physical  device.  These 
simulated  exercises  are  also  part  of  the  validation  trials  that  are  being  conducted. 
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Key  Research  Accomplishments 


•  Tele  surgery:  Communications  Latency.  Major  hospital  systems  have  sufficient 
telecommunication  bandwidth  to  perform  robotic  telesurgery  right  now. 

•  Robotic  Curriculum.  Online  curriculum  in  robotics  has  been  developed.  Psychomotor 
skills  device  has  been  developed.  Both  are  now  available  to  all  surgeons  who  are 
interested. 

•  Simulation:  Surgical  Rehearsal.  Simulation-based  surgical  rehearsal  is  currently  focused 
on  procedures  which  are  too  easy  to  offer  an  advantage  over  traditional  methods  of 
preparation. 

•  Simulation:  Military-use  Validation.  Systems  capabilities  have  been  documented  in 
published  reports  and  journals.  As  an  instructional  tool  surgeons  prefer  the  DVSS  for 
realism  and  the  dV-Trainer  for  affordability.  Surgeons  do  not  accept  the  RoSS  as  a  useful 
training  device. 
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Reportable  Outcomes 

Publications 

Tanaka,  Perez,  Truong,  &  Smith.  “From  Design  to  Conception:  An  Assessment  Device  for 
Robotic  Surgeons”,  2014  Interservice/Industry  Training  Education  and  Simulation 
(I/ITSEC)  Conference.  December  2014.  [Best  Paper  Award  for  Emerging 
Concepts/Innovative  Ideas  Track] 

Tanaka,  Graddy,  &  Smith.  “  Comparison  of  the  Usability  of  Robotic  Surgery  Simulators”,  2014 
Interservice/Industry  Training  Education  and  Simulation  (I/ITSEC)  Conference.  December 
2014. 

Smith  &  Simpson.  “  Return  on  Investment  for  Robotic  Surgical  Simulators”,  2014 

Interservice/Industry  Training  Education  and  Simulation  (I/ITSEC)  Conference.  December 
2014. 

Smith,  Truong,  &  Perez.  (2014)  Comparative  analysis  of  the  functionality  of  simulators  of  the  da 
Vinci  surgical  robot.  J  Surg  Endosc,  1-12. 

Perez,  Xu,  Chauhan,  Tanaka,  Simpson,  Abdul-Muhsin,  &  Smith.  “Impact  of  delay  on 

telesurgical  performance:  Study  on  the  dV-Trainer  robotic  simulator”.  Submission  to  Journal 
of  Urology  2014. 

Smith,  “The  Future  of  Robotic  Technology”,  Robotic  Surgery  of  the  Head  and  Neck,  Springer 
Press,  2015  (projected). 

Martino,  Siddiqui,  et  al.  “Fundamentals  of  Robotic  Gynecologic  Surgery”  Developing  a  Quality 
Improvement  Project  to  Improve  Patient  Safety”,  Society  of  Gynecologic  Oncology,  Annual 
Meeting  on  Women’s  Cancer,  March  2014. 

Smith,  Patel,  &  Satava.  “Fundamentals  of  robotic  surgery:  a  course  of  basic  robotic  surgery  skills 
based  upon  a  14-society  consensus  template  of  outcomes  measures  and  curriculum 
development”,  The  International  Journal  of  Medical  Robotics  and  Computer  Assisted 
Surgery,  October  2013.  DOI:  10.1002/rcs.l559 

Smith,  “From  FLS  to  FRS:  The  Fundamentals  of  Robotic  Surgery  are  on  their  Way”,  World 
Robotic  Gynecologic  Congress,  Chicago,  IL.  2013 

Presentations 

Truong,  Tanaka,  Simpson,  Advincula,  &  Smith.  “A  Prospective  Randomized  Controlled 
Comparative  Study  on  Surgical  Training  Methods  and  Impact  on  Surgical  Performance: 
Virtual  Reality  Robotic  Simulation  vs.  Didactic  Lectures”,  AAGL  Global  Congress  on 
Minimally  Invasive  Gynecology,  November  2014 

Smith  &  Simpson.  “Return  on  Investment  for  Robotic  Surgical  Simulators”,  AAGL  Global 
Congress  on  Minimally  Invasive  Gynecology,  November  2014 

Tanaka,  Truong,  &  Smith.  “Robotic  Surgical  Simulators:  An  Assessment  of  Usability  and 
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Conclusion 

Each  of  the  research  areas  funded  by  this  grant  has  made  significant  scientific  contributions.  The 
knowledge  gained  from  this  work  is  being  shared  through  reports  to  the  government  and  multiple 
presentations  at  both  clinical  and  simulation  conferences.  We  have  also  submitted  multiple 
papers  for  journal  publication. 

This  cooperative  agreement  was  previously  scheduled  to  end  on  August  31,  2014.  However,  we 
have  received  additional  funding  which  has  been  added  to  the  agreement.  This  new  funding  and 
the  additional  scope  extend  the  agreement  to  August  31,  2016.  Some  of  the  research  projects 
funded  in  the  original  scope  of  work  are  on-going  and  will  be  completed  during  the  period  of  the 
extension. 
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Abstract 

Background  The  implementation  of  robotic  technology  in 
minimally  invasive  surgery  has  led  to  the  need  to  develop 
more  efficient  and  effective  training  methods,  as  well  as 
assessment  and  skill  maintenance  tools  for  surgical  edu¬ 
cation.  Multiple  simulators  and  procedures  are  available  for 
educational  and  training  purposes.  A  need  for  comparative 
evaluations  of  these  simulators  exists  to  aid  users  in 
selecting  an  appropriate  device  for  their  purposes. 
Methods  We  conducted  an  objective  review  and  com¬ 
parison  of  the  design  and  capabilities  of  all  dedicated  sim¬ 
ulators  of  the  da  Vinci  robot,  the  da  Vinci  Skill  Simulator 
(DVSS)  (Intuitive  Surgical  Inc.,  Sunnyvale,  CA,  USA),  dV- 
Trainer  (dVT)  (Mimic  Technologies  Inc.,  Seattle,  WA, 
USA),  and  Robotic  Surgery  Simulator  (RoSS)  (Simulated 
Surgical  Skills,  LLC,  Williams ville,  NY,  USA).  This  pro¬ 
vides  base  specifications  of  the  hardware  and  software,  with 
an  emphasis  on  the  training  capabilities  of  each  system. 
Results  Each  simulator  contains  a  large  number  of  training 
exercises,  DVSS  =  40,  dVT  =  65,  and  RoSS  =  52  for 
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skills  development.  All  three  offer  3D  visual  images  but  use 
different  display  technologies.  The  DVSS  leverages  the  real 
robotic  surgeon’s  console  to  provide  visualization,  hand 
controls,  and  foot  pedals.  The  dVT  and  RoSS  created  sim¬ 
ulated  versions  of  all  of  these  control  systems.  They  include 
systems  management  services  which  allow  instructors  to 
collect,  export,  and  analyze  the  scores  of  students  using  the 
simulators. 

Conclusions  This  study  is  the  first  to  provide  comparative 
information  of  the  three  simulators  functional  capabilities 
with  an  emphasis  on  their  educational  skills.  They  offer 
unique  advantages  and  capabilities  in  training  robotic  sur¬ 
geons.  Each  device  has  been  the  subject  of  multiple  validation 
experiments  which  have  been  published  in  the  literature.  But 
those  do  not  provide  specific  details  on  the  capabilities  of  the 
simulators  which  are  necessary  for  an  understanding  sufficient 
to  select  the  one  best  suited  for  an  organization’s  needs. 

Keywords  Robotic  surgery  •  Robotic  simulator  • 
Training  •  Education  •  Comparative  analysis 

For  every  complex  and  expensive  system,  there  emerges  a 
need  for  training  devices  and  scenarios  which  will  assist  new 
learners  in  mastering  the  use  of  the  device  and  understand¬ 
ing  how  to  apply  it  with  value.  In  laparoscopic  surgery, 
simulators  have  played  an  important  role  in  improving  the 
practice  of  surgery  over  the  last  20  years  [1,  2].  The  same 
trends  and  values  will  likely  apply  to  robotic  surgery  with 
the  increased  use  of  robotic  technology  for  a  growing  variety 
of  minimally  invasive  surgical  procedures.  The  complexity, 
criticality,  and  cost  associated  with  the  effective  application 
of  the  da  Vinci  surgical  robot  have  stimulated  the  com¬ 
mercial  creation  of  simulators,  which  replicate  the  opera¬ 
tions  of  this  robot.  The  objective  of  this  paper  was  to  provide 
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Fig.  1  Simulators  of  the  da  Vinci  surgical  robot 


comparative  data  on  the  functionality  of  the  three  com¬ 
mercially  available  robotic  simulators  as  shown  in  Fig.  1 : 

•  da  Vinci  Skill  Simulator  (Intuitive  Surgical  Inc., 
Sunnyvale,  CA,  USA); 

•  dV-Trainer  (Mimic  Technologies,  Inc.,  Seattle,  WA, 
USA);  and 

•  RoSS  (Simulated  Surgical  Skills  LLC,  Williamsville, 
NY,  USA). 

Each  of  these  possesses  unique  traits  which  make  them 
valuable  solutions  for  different  types  of  users  and  learning 
environments.  This  report  is  on  the  first  of  a  three  part 
comparative  analysis  of  these  devices.  The  first  examines 
the  functionality  of  each  of  the  simulators  and  illustrates 
these  capabilities  side-by-side  for  ease  of  evaluation  by 
potential  users  of  each  device.  The  second  is  a  subjective 
usability  evaluation  of  the  simulators  on  similar  exercises 
by  novice  (medical  students),  intermediate  (residents  and 
fellows),  and  expert  (attending  surgeons)  subjects.  The 
third  is  measure  of  the  degree  to  which  each  simulator 
improves  the  actual  robotic  skills  of  a  subject  who  is 
engaged  in  a  two  months  training  program  with  the  device. 
This  paper  presents  the  results  of  the  first  study  defining  the 
functionality  of  the  devices. 


to  objectively  evaluate  and  comment  on  each  device 
without  undue  influence  from  the  manufacturers.  Each 
simulator  company  was  aware  of  the  comparison  project 
and  provided  information  on  their  device  in  response  to 
queries  by  our  researchers,  as  noted  below.  We  began  by 
reviewing  the  users’  manuals  for  the  devices  to  collect 
details  about  each  system  [3-5].  We  then  interviewed 
representatives  of  each  of  the  manufacturing  companies  for 
additional  functional  details.  Finally,  we  performed  our 
own  experiments  with  each  device  to  identify  important 
comparative  features  across  all  devices. 

We  conducted  a  systematic  literature  review  of  all  three 
simulators.  The  PubMed  database  of  medical  research  was 
searched  for  all  references  to  the  devices  through  March 
2013.  References  from  retrieved  articles  were  reviewed  to 
broaden  the  search.  The  data  extracted  from  these  studies 
include  training  exercise  modules,  scoring  systems,  costs, 
educational  impact,  and  validation  methods.  We  identified 
45  studies  investigating  simulation  in  robotic  surgery. 

Finally,  we  submitted  our  comparative  data  on  the  sys¬ 
tems  to  the  manufacturers  of  the  devices  to  verify  the 
accuracy  of  the  information.  Each  company  verified  that 
the  data  presented  in  this  analysis  were  accurate. 

Results 


Materials  and  methods 

Our  department  purchased  each  of  the  simulator  devices 
which  are  being  evaluated  in  these  studies.  This  allowed  us 


Each  of  these  devices  is  manufactured  by  a  different  com¬ 
pany  and  provides  a  unique  hardware  and  software  solution 
for  training  and  surgical  rehearsal.  The  general  features  and 
capabilities  of  each  are  summarized  in  Table  1. 
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Table  1  Robotic  simulator  feature  comparison 

Features 

DVSS 

dV-Trainer 

RoSS 

System  manufacturer 

Intuitive  Surgical  Inc. 

Mimic  Technologies  Inc. 

Simulated  Surgical  Systems  LLC 

Specifications 

Depth  7" 

Depth  36" 

Depth  44" 

(simulator  only) 

Height  25" 

Width  23" 

120  or  240  V  power 

Height  26" 

Width  44" 

120  or  240  V  power 

Height  11" 

Width  45" 

120  or  240  V  power 

Specifications 

Depth  41" 

Depth  36" 

Depth  44" 

(complete  system  as 
shown  in  Fig.  1) 

Height  65" 

Width  40" 

120  or  240  V  power 

Height  59" 

Width  54" 

120  or  240  V  power 

Height  77" 

Width  45" 

120  or  240  V  power 

Visual  resolution 

VGA  1,024  x  768 

VGA  1,024  x  768 

VGA  640  x  480 

Components 

Customized  computer  attached  to 
da  Vinci  surgical  console 

Standard  PC,  visual  system  with  hand 
controls,  foot  pedals 

Single  integrated  custom 
simulation  device 

Support  equipment 

da  Vinci  Si  surgical  console, 
custom  data  cable 

Adjustable  table,  touch  screen  monitor, 
keyboard,  mouse,  protective  cover,  custom 
shipping  container 

USB  adapter,  keyboard,  mouse 

Exercises 

40  simulation  exercises 

65  simulation  exercises 

52  simulation  exercises 

Optional  software 

PC -based  simulation  management 

Mshare  curriculum  sharing  web  site 

Video  and  haptics-based 
procedure  exercises  (HoST) 

Scoring  method 

Scaled  0-100  %  with  passing 
thresholds  in  multiple  skill  areas 

Proficiency-based  point  system  with  passing 
thresholds  in  multiple  skill  areas 

Point  system  with  passing 
thresholds  in  multiple  skill 
areas 

Student  data 

Custom  control  application  for 

Export  student  data  to  delimited  data  file  and 

Export  student  data  to  delimited 

management 

external  PC.  Export  via  USB 
memory  stick 

graphical  reports 

data  file 

Curriculum 

customization 

None 

Select  any  combination  of  exercises.  Set 
passing  thresholds  and  conditions 

Select  specifically  grouped 
exercises.  Set  passing 
thresholds 

Administrator 

Create  student  accounts  on 

Create  student  accounts.  Customize 

Create  student  accounts. 

functions 

external  PC.  Import  via  USB 
memory  stick 

curriculum 

Customize  curriculum 

System  setup 

None 

Calibrate  controls 

Calibrate  controls 

System  security 

Student  account  ID  and  password 

PC  password,  Administrator  password, 

Student  account  ID,  and  password 

PC  password,  Administrator 
password,  student  account  ID, 
and  password 

Simulator  base  price 

$85,000 

$99,200 

$126,000 

Support  equipment 
price 

$500,000 

$9,800 

$0 

Total  functional  price 

$585,000 

$109,000 

$126,000 

Data  are  for  simulator  configurations  available  as  of  December  2013 


Features  and  capabilities 

Da  Vinci  Skill  Simulator  (DVSS)  ( Intuitive  Surgical  Inc.) 

The  DVSS  consists  of  a  customized  computer  package  that 
attaches  to  the  back  of  the  surgeon’s  console  of  an  actual  da 
Vinci  Si  robot.  This  simulator  connects  to  the  surgeon’s 
console  via  a  single  fiber  optic  networking  cable  identical 
to  that  used  to  connect  the  components  of  the  actual  robotic 
surgical  system. 


Advantages 

Attached  simulators  of  this  type  are  usually  referred  to  as 
“embedded  trainers”  because  they  take  advantage  of  the 
equipment  that  has  already  been  constructed,  purchased,  and 
installed  for  the  use  of  the  real  system.  These  kinds  of  sim¬ 
ulators  are  especially  common  in  military  facilities  which 
face  limited  space  and  weight  constraints.  They  can  signifi¬ 
cantly  reduce  the  hardware  that  must  be  purchased  solely  for 
simulation  purposes.  The  U.S.  Navy  uses  these  kinds  of 


Springer 


Surg  Endosc 


simulators  aboard  ships  to  reduce  weight  and  space 
requirements,  enabling  them  to  train,  while  the  ship  is  at  sea. 

Another  significant  advantage  of  an  attached  simulator 
is  that  it  allows  the  trainee  to  use  the  actual  controls  from 
the  real  system  to  drive  the  simulator.  This  ensures  that  the 
training  experience  is  almost  identical  in  feel  to  the  real 
system,  which  can  contribute  to  higher  transfer  of  skills 
from  the  training  sessions  to  the  real  system.  Additionally, 
this  minimizes  the  amount  of  time  spent  for  learning  the 
unique  functionalities  of  the  simulator  device  and  allows 
the  trainee  to  focus  the  majority  of  his/her  learning  expe¬ 
rience  on  skills  acquisition  and  proficiency  development. 
Finally,  there  is  the  cost  advantage  for  the  simulator  device 
itself.  Because  much  of  the  hardware  and  software 
expenses  are  already  embedded  in  the  real  system,  the 
simulator  can  be  very  economical  to  purchase. 

Disadvantages 

Attached  simulators  like  the  DVSS  also  come  with  inherent 
disadvantages  to  balance  their  positive  traits. 

The  largest  drawback  is  the  availability  and  accessibility  of 
a  simulator  which  requires  the  real  robotic  system.  An 
attached  DVSS  simulator  cannot  be  used  without  access  to  an 
actual  surgeon’ s  console  and  therefore  is  only  functional  when 
the  robotic  system  is  not  in  surgery.  This  implies  that  the 
trainee  would  only  be  able  to  use  the  simulator  outside  of 
normal  operating  room  working  hours  and  would  need 
logistical  access  to  the  robot  and  the  simulator,  da  Vinci  robots 
are  expensive  devices  and  hospitals  typically  attempt  to 
maximize  use  of  in  order  to  recoup  their  investment.  In  a  very 
active  surgical  hospital,  it  can  be  difficult  to  obtain  access  to  a 
surgeon’s  console  to  support  training  with  this  simulator. 

The  DVSS  is  designed  to  connect  to  the  surgeon’s 
console  using  the  same  networking  cable  that  connects  the 
major  robotic  components.  This  makes  the  attachment  and 
set-up  process  very  easy  for  clinicians  to  master.  However, 
it  also  means  that  the  DVSS  can  only  be  used  with  the  Si 
model  surgeon’s  console.  The  previous  S  and  Standard 
models  use  a  different  set  of  cables,  which  are  not  com¬ 
patible  with  the  simulator. 

Similar  to  the  military’s  experience  with  embedded  and 
attached  simulators,  heavy  usage  of  the  DVSS  comes  with 
a  corresponding  heavy  use  of  the  surgeon’s  console.  The 
Army  and  Navy  have  discovered  that  these  types  of  sim¬ 
ulators  put  more  usage  hours  on  real  equipment  controls 
which  lead  to  more  maintenance  costs  for  those  devices. 
Given  the  possibility  of  regular  and  continuous  simulation 
training  with  such  device,  in  addition  to  actual  surgical 
usage,  the  real  equipment  may  experience  usage  rates  that 
are  many  times  higher  than  normal  for  the  equipment. 
Since  the  da  Vinci  systems  operate  under  a  maintenance 
contract  that  covers  most  service  costs,  the  additional  costs 


of  maintenance  are  not  born  by  the  hospital  owner  but  by 
the  equipment  vendor.  The  primary  impact  to  the  owner 
would  only  be  in  availability  for  both  real  surgeries  and 
training  events  due  to  increased  maintenance. 

dV-Trainer  ( Mimic  Technologies  Inc.) 

The  dV-Trainer  is  a  separate,  stand-alone  simulator  of  the  da 
Vinci  robot.  The  surgeon’s  console,  controls,  and  vision  cart 
are  mimicked  in  hardware,  while  a  3D  software  model  repli¬ 
cates  the  functions  of  the  robotic  arms  and  the  surgical  space. 

Mimic  Technologies  also  developed  the  core  simulator 
software  for  the  DVSS  and  used  the  same  package  in 
version  1.0  of  their  own  dV-Trainer.  As  a  result,  the 
exercises  in  the  DVSS  and  version  1.0  of  the  dV-Trainer 
are  nearly  identical.  The  current  version  2.2  of  the  dV- 
Trainer  has  a  number  of  new  exercises  which  are  not  found 
in  the  DVSS,  and  the  graphics  have  been  upgraded  so  the 
visual  presentation  is  no  longer  identical.  The  differences 
in  visual  presentation  can  be  seen  in  Fig.  3  and  4. 

The  dV-Trainer  consists  of  three  major  pieces  of 
equipment  and  a  number  of  smaller  support  pieces.  The 
largest  pieces  are  the  “Phantom”  hood  which  replicates  the 
vision  and  hand  controls  of  the  da  Vinci  surgeon’s  console, 
the  foot  pedals  of  the  surgeon’s  console,  and  a  high-per¬ 
formance  desktop  computer  which  generates  the  3D  ima¬ 
ges  and  calculates  the  interactions  with  the  surgeon’s 
controls.  Smaller  support  equipment  includes  a  touch 
screen  monitor,  keyboard,  and  mouse  to  enable  an 
instructor  to  guide  the  student  through  exercises  and  allow 
an  administrator  to  manage  the  data  that  are  collected. 

Because  the  dV-Trainer  replicates  both  the  hardware 
and  software  of  the  da  Vinci  robot,  it  is  a  much  larger 
system  than  the  DVSS  alone,  though  smaller  than  a  real 
surgeon’s  console  with  the  DVSS  attached.  It  has  the 
advantage  of  providing  a  training  system  that  is  completely 
independent  of  the  need  for  any  piece  of  the  real  surgical 
robot.  The  simulator  can  be  configured  to  imitate  either  the 
S  or  the  Si  model  of  the  da  Vinci  robot. 

The  disadvantage  of  this  kind  of  system  is  that  the 
simulated  hardware  is  different  than  the  real  equipment  and 
does  not  exactly  replicate  the  feel  of  the  real  robotic 
equipment.  The  dV-Trainer  uses  its  own  unique  hand 
controls  which  are  connected  to  three  cables  for  measuring 
movement,  rather  than  the  more  precise  arms  that  are  used 
in  the  da  Vinci  robot.  The  dV-Trainer  foot  pedals  look  and 
function  almost  identically  to  the  robotic  foot  pedals. 

Robotic  Surgery  Simulator  ( Simulated  Surgical  Systems 
LLC) 

The  RoSS  is  also  a  complete,  stand-alone  simulator  of  the 
da  Vinci  robot.  This  device  is  designed  as  a  single  piece  of 
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hardware  that  has  a  similar  appearance  to  the  surgeon’s 
console  of  the  robot.  The  hardware  device  includes  a  single 
3D  computer  monitor,  hand  controls  that  are  modified 
commercial  force  feedback  devices,  pedals  that  replicate 
either  the  S  or  the  Si  model  of  the  da  Vinci  robot,  and  an 
external  monitor  for  the  instructor.  The  simulator  can  be 
configured  to  imitate  either  the  S  or  the  Si  model  of  the  da 
Vinci  robot. 

The  hand  controls  are  modified  SensAble  Omni  Phan¬ 
tom™,  force  feedback,  3D  space  controllers  (3D  Systems 
Inc.,  Rock  Hill,  SC,  USA).  These  devices  have  a  much 
smaller  range  of  motion  than  the  controllers  on  the  da  Vinci 
robot,  so  require  more  frequent  clutching  than  the  actual 
robot.  The  3D  image  is  generated  by  a  single  computer 
monitor  with  polarized  glasses,  which  generates  a  visual 
scene  with  less  depth  of  field  than  the  actual  robot. 

The  company  has  developed  a  set  of  3D  virtual  exercises 
that  are  unique  from  those  found  in  both  of  the  other  simula¬ 
tors.  They  also  provide  optional  video-based  surgical  exer¬ 
cises,  called  HoST  modules,  in  which  the  user  is  guided 
through  the  movements  necessary  to  complete  an  actual  sur¬ 
gical  procedure.  At  this  writing,  these  modules  are  available 
for  radical  prostatectomy,  hysterectomy,  and  cystectomy. 
These  guided  videos  take  advantage  of  the  force  feedback 
capabilities  of  the  hand  controllers  to  push  and  pull  the  stu¬ 
dent’  s  hands  to  follow  the  simulated  instruments  on  the  screen. 
They  require  the  student  to  perform  specific  movements 
accurately  during  the  video  before  the  operation  will  proceed. 

Exercise  modules 

Each  simulator  allows  an  administrator  or  instructor  to 
manage  and  organize  student  performance  according  to 
unique  login  credentials  for  the  student.  Alternatively,  they 
all  have  a  universal  “guest”  account  to  make  the  system 
accessible  to  anyone  but  without  the  ability  to  uniquely 
identify  and  track  the  performance  of  a  specific  student. 

Once  logged  into  each  system,  the  instructor  or  the 
student  navigates  the  instructional  materials  using  the 
menu  systems  illustrated  in  Fig.  2.  Since  the  intuitive  skills 
simulator  (DVSS)  and  the  Mimic  dV-Trainer  provide  very 
similar  exercises  and  organizations,  the  navigation  through 
the  exercises  is  similar  in  form,  though  different  in  visual 
appearance.  The  RoSS  simulator  uses  a  very  unique  arced 
orbital  menu  for  progressing  through  exercises. 

Each  simulator  provides  on- system  instructions  for 
every  exercise  in  the  form  of  textual  documents  and  video 
demonstrations  with  spoken  audible  instructions. 

Dvss 

The  DVSS  contains  40  exercises  organized  into  nine  cat¬ 
egories  (Table  2).  These  begin  with  introductory  video  and 


audio  instructions  on  how  to  use  the  robotic  equipment  and 
move  through  progressively  more  difficult  skills  (Table  3). 

To  prepare  the  student  for  success  in  each  exercise,  the 
simulator  offers  written  instructions  on  the  objective  of 
each  exercise  prior  to  performance.  There  is  also  a  narrated 
video  of  an  instructor  performing  the  exercise  while 
explaining  the  necessary  steps. 

Upon  completion  of  each  exercise,  the  system  auto¬ 
matically  proceeds  to  a  scoreboard  showing  the  student’s 
performance  on  the  exercise.  Details  on  the  scoring  sys¬ 
tems  of  each  simulator  are  discussed  later  in  the  article. 

Figure  3  presents  screenshots  of  some  of  the  key  exercises 
in  the  simulator.  These  include  the  Peg  Board,  Ring  Walk, 
Energy  Dissection,  and  Interrupted  Suturing  exercises.  The 
suturing  exercises  on  this  simulator  were  developed  by 
Simbionix  USA  Inc.  (Cleveland,  OH)  for  integration  into  the 
DVSS.  This  expansion  of  the  system  demonstrates  the  ability 
of  the  simulator  platform  to  blend  together  exercises  and 
scoring  systems  created  by  multiple  independent  vendors. 

dV-Trainer 

Most  of  the  simulation  software  for  Intuitive’s  DVSS  was 
developed  by  Mimic  Technologies.  Therefore,  version  1.0  of  the 
DVSS  and  the  dV-Trainer  contained  nearly  identical  exercises, 
closely  matching  menu  systems,  and  identical  scoring  mecha¬ 
nisms.  However,  over  time  the  two  sets  of  software  have  diverged, 
and  the  current  versions  of  the  simulators  differ  in  functionality 
and  appearance.  The  current  version  of  the  dV-Trainer  (v  2.2) 
contains  65  exercises  organized  into  ten  categories. 

Though  many  of  the  exercises  are  identical  between  the 
DVSS  and  the  dV-Trainer,  the  graphics  resolution  and 
details  have  been  improved  in  version  2.2  of  the  dV- 
Trainer  software.  Since  this  system  is  driven  by  a  com¬ 
mercial  PC,  which  can  easily  be  upgraded,  it  is  possible  for 
the  hardware  and  software  to  evolve  as  newer  computer 
technologies  are  available. 

Just  as  with  the  DVSS,  the  dV-Trainer  simulator  offers 
written  instructions  on  the  objective  of  each  exercise  prior 
to  performance.  There  is  also  a  narrated  video  of  an 
instructor  performing  the  exercise  while  explaining  the 
necessary  steps.  Upon  completion  of  each  exercise,  the 
system  automatically  proceeds  to  a  scoreboard  showing  the 
student’s  performance  on  the  exercise. 

Figure  4  presents  screenshots  of  some  of  the  key  exer¬ 
cises  in  the  dV-Trainer  simulator.  These  include  the  Peg 
Board,  Match  Board,  Tubal  Anastomosis,  and  Energy 
Switching  exercises. 

RoSS 

The  RoSS  simulator  contains  52  unique  exercises,  orga¬ 
nized  into  five  categories,  and  arranged  from  introductory 
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Da  Vinci  Skills  Simulator  dV-Trainer 


RoSS 


Fig.  2  Comparative  simulator  exercise  menus 


Table  2  DVSS  exercise  categories 


Surgeon  console 
overview 

An  introduction  to  the  controls  of  the  da  Vinci 
robot 

Endowrist 

Basic  hand  movements  and  usage  of  the  wristed 

manipulation  1 

instruments 

Camera  and 

Basic  foot  clutching  for  both  the  camera  and  the 

clutching 

third  arm 

Endowrist 

Intermediate  use  of  the  hands  and  wristed 

manipulation  2 

instruments 

Energy  and 

Use  of  the  energy  pedals  and  associated 

dissection 

instruments 

Needle  control 

Focused  exercises  for  dexterous  manipulation  of 
a  curved  surgical  needle 

Needle  driving 

Repetitive  exercises  for  needle  driving 

Games 

Challenging  and  entertaining  game 
environments  to  apply  the  skills  learned 

Suturing  skills 

Suturing  exercises  with  needle,  following 
suture,  knot-tying,  and  tissue  closure 

to  more  advanced  (Table  4),  just  as  in  the  other  two  sim¬ 
ulators.  The  RoSS  system  of  exercises  is  unique  in  that  they 
list  fewer  exercises  but  provide  three  different  difficulty 
levels  for  most  of  them  where  each  level  is  actually  a 
unique  exercise. 

Similar  to  the  other  simulators,  the  RoSS  includes  a 
narrated  video  showing  an  instructor  performing  the  exer¬ 
cise.  Upon  completion  of  an  exercise,  the  simulator  auto¬ 
matically  proceeds  to  the  scoreboard  for  the  exercise. 

The  RoSS  contains  a  unique  capability  that  is  not  found  in 
either  of  the  other  simulators  called  “Hands-on  Surgical 
Training”  or  “HoST.”  This  is  an  integration  of  surgical  skills 
exercises  with  a  video  of  an  actual  surgery.  Videos  of  actual 
surgical  procedures  play  in  the  surgeon’s  visual  space,  over¬ 
laid  with  animated  icons,  which  instruct  the  student  to  perform 
specific  actions  during  the  progression  of  the  surgery  video. 


Table  3  dV-Trainer  exercise  categories 


Surgeon  console 
overview 

An  introduction  to  the  controls  of  the  da  Vinci 
robot 

Endowrist 

Basic  and  intermediate  use  of  the  hand 

manipulation 

controllers  and  wristed  instruments 

Camera  and 

Basic  foot  clutching  for  both  the  camera  and  the 

clutching 

third  arm 

Energy  and 

Use  of  the  energy  pedals  and  associated 

dissection 

instruments 

Needle  control 

Focused  exercises  for  dexterous  manipulation  of 
a  curved  surgical  needle 

Needle  driving 

Repetitive  exercises  for  needle  driving 

Troubleshooting 

Introduction  to  error  recovery  on  the  da  Vinci 
robot 

Games 

Challenging  and  entertaining  game 
environments  to  apply  the  skills  learned 

Suturing  skills 

Suturing  exercises  with  needle,  following  suture, 
knot-tying,  and  tissue  closure 

RTN 

VR  exercises  specifically  build  to  match 
physical  devices  in  use  by  the  research  training 
network  of  sites  led  by  Lehigh  Valley  Hospital 

The  necessary  actions  are  prompted  with  audio  instructions. 
For  the  HoST  exercise  to  progress,  the  student  must  perform 
the  specific  actions  at  specific  times.  The  simulator  will  pause 
the  video  and  allow  the  student  to  repeat  the  action  until  it  is 
performed  as  required  by  the  instructions. 

The  hand  controllers  of  the  RoSS  simulator  are  modified 
versions  of  a  commercially  available  3D  haptic  input 
device  called  the  Omni  Phantom™.  This  product  uses 
internal  motors  and  gears  to  apply  haptic  feedback  to  the 
hand  movements  of  the  user.  For  the  HoST  exercises,  the 
simulator  uses  this  capability  to  move  the  student’s  hands 
in  sync  with  the  movements  of  the  surgeon’s  instruments  in 
the  master  video. 
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Fig.  3  Selected  DVSS  exercise 

images  Da  Vinci  Skills  Simulator 


Fig.  4  Selected  dV-Trainer 

exercise  images  dV-Trainer 
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Table  4  RoSS  exercise  categories 


Orientation 

module 

Introduction  to  the  surgeon  controls  of  the  da 
Vinci  robot 

Motor  skills 

Development  of  precise  controls  of  the 
instruments,  including  spatial  awareness 

Basic  surgical 
skills 

Instruction  on  handling  a  needle,  using 
electrocautery  pedals  and  instruments,  and  the 
use  of  scissors  on  the  robot 

Intermediate 
surgical  skills 

Control  of  the  fourth  arm,  blunt  tissue 
dissection,  and  vessel  dissection 

Hands-on  surgical 
training 

Video  and  haptic-guided  instruction  through 
specific  surgical  procedures 

Figure  5  provides  screenshots  of  the  motor  skills  ball 
placement,  intermediate  vessel  dissection,  4th  arm  tissue 
retraction,  and  HoST  radical  prostatectomy. 

Proficiency  scoring  system 

Each  of  the  three  simulators  provides  a  different  scoring 
method.  All  three  use  the  host  computer  to  collect  data  on 
the  performance  of  the  student  at  the  controls  in  multiple 
performance  areas.  With  this  data,  they  provide  a  score  for 
specific  performance  traits,  as  well  as  combining  all  of 
these  into  a  single  composite  score  of  performance  for  the 
entire  exercise.  The  algorithm  used  to  create  this  composite 
score  is  described  in  the  user’s  manuals  of  each  of  the 
simulators.  Examples  of  each  of  these  scoreboards  are 
shown  in  Fig.  6. 


In  addition  to  the  objective  metrics  that  can  be  collected 
by  the  computer,  the  developers  of  each  simulator  have 
been  challenged  to  provide  thresholds,  which  indicate 
whether  the  student’s  score  is  considered  a  “passing”  or 
“failing”  performance.  All  three  have  identified  threshold 
scores,  which  would  indicate  acceptable  and  warning 
scoring  levels.  These  are  commonly  interpreted  as  “pass¬ 
ing”  (above  acceptable  threshold)  and  “failing”  (below 
warning  threshold),  with  a  “warning”  area  between  the  two 
thresholds.  These  thresholds  create  green,  yellow,  and  red 
performance  areas,  which  can  be  used  to  visually  com¬ 
municate  the  quality  of  the  student’s  performance  in  each 
area  of  measurement.  Each  simulator  also  provides  a  single 
composite  score  for  the  entire  exercise. 

Each  of  the  simulators  gives  the  student  a  single  overall 
score  for  performance  on  an  exercise.  To  achieve  this,  an 
algorithm  was  needed  to  combine  very  different  types  of 
metrics.  For  example,  the  number  of  seconds  to  complete 
an  exercise  needs  to  be  combined  with  milliliters  of  blood 
loss,  centimeters  of  instrument  movement,  number  of 
instrument  collisions,  and  other  similarly  varied  metrics. 
As  in  most  educational  environments,  this  is  achieved  by 
converting  each  metric  into  a  score,  which  falls  between 
some  defined  minimum  and  maximum  value.  Most  people 
understand  this  concept  from  their  academic  experience  in 
which  all  assignments  were  graded  in  the  range  from  0  to 
100  %  or  between  0  points  and  the  maximum  total  points 
for  all  assignments.  These  normalizations  make  it  possible 
to  create  a  single  composite  score  of  the  student’s 


Fig.  5  Selected  RoSS  exercise 
images 


RoSS 
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Fig.  6  Example  scoreboards  from  each  simulator 

performance  across  multiple  assignments.  This  same 
approach  has  been  used  in  the  simulators,  where  the 
resulting  composite  metric  may  be  a  total  point  score  or  a 
percentage. 

The  simulator  manufacturers  work  with  robotic  sur¬ 
geons  to  establish  the  relative  values  of  each  measure  used 
in  the  composite  score,  just  as  they  did  for  the  threshold 
levels  described  earlier.  Because  these  evaluations  are  the 
opinions  of  the  individuals  who  collaborated  with  the 
company  on  the  development  of  the  system.  The  dV- 
Trainer  and  the  RoSS  both  provide  the  ability  for  a  system 
administrator  to  adjust  these  levels  to  meet  the  needs  of 
unique  curriculum,  courses,  and  students  being  evaluated. 

DVSS 

The  DVSS  performance  scoring  method  has  a  number  of 
metrics  which  are  applied  to  every  exercise,  and  others 
which  are  only  used  for  exercises  in  which  they  are  rele¬ 
vant.  Table  5  presents  the  metrics  which  are  applicable  to 
all  exercises.  For  details  on  the  more  specialized  metrics, 
the  reader  may  consult  the  user’s  manual  for  the  simulator. 

Because  the  DVSS  is  a  closed,  turnkey  system  with  an 
ease  of  use  similar  to  the  actual  surgical  robot,  most  of  the 
data  displays,  and  threshold  adjustments  found  in  the  other 
simulators  are  not  available  in  this  device.  Simulator  set¬ 
tings  are  determined  by  the  manufacturer  and  cannot  be 
changed  by  the  user. 

dV -Trainer 

Originally,  the  DVSS  and  the  dV-Trainer  shared  the  same 
scoring  method,  but  more  recent  versions  of  the  dV- 
Trainer  offer  both  this  original  “version  1.0”  scoring 
method,  as  well  as  a  new  “version  2.0”  method  based  on 
the  proficiency  measured  from  experienced  surgeons.  The 
skills  measured  are  the  same  (Table  3),  but  the  interpre¬ 
tation  of  those  into  a  score  is  different.  The  instructor  can 


Table  5  DVSS  and  dV-Trainer  scoring  method 


Overall  score 

Time  to  complete 

Economy  of 
motion 


Composite  evaluation  of  the  exercise 
performance 

Number  of  seconds  to  complete  the  exercise 

Number  of  centimeters  of  instrument  tip 
movement 


Instrument  Number  of  times  that  the  instruments  touched 

collisions  each  other 


Excessive 
instrument  force 

Instrument  out  of 
view 

Master  workspace 
range 

Drops 


Number  of  seconds  that  excessive  robotic  force 
was  applied  against  objects  in  the  environment 

Number  of  centimeters  that  an  instrument  tip 
moved  outside  of  the  viewing  area 

Radius  in  centimeters  than  contains  the 
movement  of  the  instrument  tips 

Number  of  objects  dropped  from  the  grasp  of 
the  instruments 


select  the  preferred  scoring  method  for  each  curriculum 
that  is  constructed  in  the  dV-Trainer.  The  newer  scoring 
method  uses  total  points  earned  rather  than  percentages. 
The  passing  and  warning  thresholds  can  be  adjusted  by 
the  administrator. 


RoSS 

The  principles  behind  the  scoring  system  on  the  RoSS  are 
the  same  as  those  for  the  DVSS  and  the  dV-Trainer. 
However,  most  of  the  metrics  collected  are  different.  The 
standard  measurements  are  shown  in  Table  6. 

Like  each  of  the  other  simulators,  there  are  multiple 
displays  of  the  performance  data  for  a  student.  The  initial 
display  presented  at  the  completion  of  an  exercise  shows  a 
horizontal  bar,  which  is  colored  green,  yellow,  or  red  to 
indicate  passing  or  failing.  The  magnitude  of  the  bar  is  a 
rough  measure  of  the  quality  of  performance  (Fig.  6). 
Additional  displays  show  the  numeric  score  and  its  relative 
position  to  a  passing  threshold. 
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Table  6  RoSS  scoring  method 


Overall  score 

Camera  usage 

Left  tool  grasp 

Left  tool  out  of 
view 

Number  of  errors 

Right  tool  grasp 

Right  tool  out  of 
view 

Time 

Tissue  damage 

Tool-Tool 

collision 


Composite  evaluation  of  the  exercise  performance 
Optimal  movement  of  camera 
Optimal  number  of  tool  grasps  with  left  hand  tool 
Distance  left  hand  tool  is  out  of  view 

Number  of  collision  or  drop  errors  in  an  exercise 

Optimal  number  of  tool  grasps  with  right  hand 
tool 

Distance  right  hand  tool  is  out  of  view 

Time  to  complete  the  exercise 

Number  of  times  that  instruments  damaged  tissue 
with  excessive  force  or  unnecessary  touches 

Number  of  times  tools  touched  each  other 


System  administration 

All  of  the  simulators  contain  system  configuration  and 
student  management  functions,  which  require  a  special 
administrator  account  to  access  and  modify.  These  allow 
instructors  to  create  curriculum  and  scoring  methods, 
which  are  unique  to  the  lessons  they  are  offering.  They  also 
allow  an  instructor  or  administrator  to  create  new  student 
accounts  and  export  student  scores  for  evaluation  and 
analysis  outside  of  the  simulator  device.  Some  course 
instructors  use  this  capability  to  create  custom  performance 
reports  for  students  who  attend  the  courses. 

DVSS 

For  the  DVSS,  most  of  the  administrator  functionality  is 
fixed  within  the  delivered  system.  The  administrator  can 
create  specific  user  profiles  for  the  simulator  using  a  ded¬ 
icated  program  on  a  separate  external  PC.  This  program, 
the  “DVSS  Manger”,  allows  the  administrator  to  create  a 
profile  for  the  user.  The  profile  can  then  be  loaded  onto  a 
USB  memory  stick  and  inserted  into  the  USB  port  on  the 
DVSS.  The  simulator  will  automatically  read  this  data  in 
and  display  the  user  names  at  the  login  screen. 

Similarly,  the  USB  memory  stick  can  be  inserted  into 
the  DVSS,  and  the  performance  data  collected  from  exer¬ 
cises  performed  by  each  user  will  be  automatically  loaded 
onto  the  USB  stick.  This  stick  can  then  be  inserted  in  the 
PC,  and  the  data  will  be  loaded  into  the  management 
software  on  the  external  PC  and  exported  to  a  delimited  file 
for  formatting  and  analysis  in  a  spreadsheet  program. 

The  entire  transfer  process  is  automated  and  the  contents 
of  the  USB  stick  are  completely  erased  and  reloaded  each 
time.  The  stick  cannot  safely  be  used  for  any  purpose  other 
than  as  the  transfer  mechanism  between  the  two  devices. 


This  method  is  meant  to  create  an  ease  of  use  similar  to  the 
real  robot. 

dV -Trainer 

The  administrator  on  a  dV-Trainer  has  the  ability  to  create 
new  user  accounts,  specify  S  or  Si  representation,  create 
new  curriculum,  set  passing  thresholds,  and  export  user 
data  for  analysis. 

The  simulator  contains  65  exercises,  any  combination  of 
which  can  be  organized  into  a  curriculum  for  a  specific 
course.  The  administrator  creates  the  new  curriculum  name 
and  then  adds  each  exercise  that  should  be  part  of  the 
curriculum.  This  set  of  exercises  can  be  organized  into 
phases  or  folders  to  match  the  course  that  is  being  taught. 
For  example,  an  instructor  may  have  a  curriculum  that 
consists  of  a  warm-up  with  easy  exercises,  pre-course 
evaluations,  and  post-course  evaluations.  These  would 
appear  as  three  separate  sections  within  the  curriculum. 

The  administrator  can  export  data  from  the  simulator 
according  to  multiple  criteria.  The  export  may  include  all 
of  the  data  on  the  machine,  or  subsets  defined  by  the  unique 
user  ID,  date  range,  completion  status,  or  a  specific 
exercise. 

The  capabilities  provided  for  an  administrator  of  the  dV- 
Trainer  are  significantly  more  robust  than  those  available 
on  the  other  two  simulators. 

RoSS 

The  RoSS  administrator  account  is  used  to  create  student 
accounts.  Each  user  can  then  be  assigned  a  specific  subset 
of  the  entire  simulator  curriculum. 

For  the  RoSS  system,  the  administrator  can  assign 
portions  of  the  curriculum  hierarchy,  which  are  applicable 
to  a  specific  user.  The  curriculum  is  organized  such  that 
customization  consists  of  selective  subsets  of  the  hierarchy 
of  exercises,  rather  than  the  ability  to  select  specific  exer¬ 
cises  in  unique  combinations. 

The  administrator  can  also  edit  the  passing  thresholds 
for  each  exercise.  This  allows  a  site  to  create  curriculum, 
which  is  considered  passing  for  practitioners  at  different 
levels,  such  as  medical  students,  residents,  attending,  and 
specialists. 

The  scores  can  be  exported  as  individual  delimited  data 
files  for  each  student  account.  These  can  then  be  removed 
from  the  system  for  analysis  and  recording. 

Validation  of  devices 

Validation  studies  serve  to  determine  whether  a  simulator 
can  actually  teach  or  assess  what  it  is  intended  to  teach  or 
assess.  In  medical  simulation,  there  are  generally  accepted 
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Table  7  Validation  of  robotic  surgical  simulators 


Validation 

DVSS 

dV-Trainer 

RoSS 

Face:  subjective 

Hung  [7] 

Lendvay  [10] 

Seixas- 

realism  of  the 
simulator 

Kelly  [8] 

Liss  [9] 

Kenney  [11] 
Sethi  [12] 

Perrenot  [13] 

Korets  [14] 

Lee  [15] 
Schreuder 
[16] 

Mikelus 

[17] 

Stegemann, 

[18] 

Content:  judgment  of 

Hung  [7] 

Kenney  [11] 

Seixas- 

appropriateness  as 

Hung 

Sethi  [12] 

Mikelus 

a  teaching  modality 

[19] 

Kelly  [8] 

Liss  [9] 

Perrenot  [13] 

Lee  [15] 

[17] 

Colaco,  [20] 

Construct:  able  to 
distinguish 
experienced  from 
inexperienced 
surgeon 

Hung  [7] 
Kelly  [8] 

Liss  [9] 
Finnegan 
[21] 

Kenney  [11] 
Perrenot  [13] 

Korets  [14] 

Lee  [15] 
Schreuder 
[16]  Connolly 

[22]  Lendvay 

[23] 

Raza  [24] 

Concurrent:  extent  to 

Hung  [19] 

Perrenot  [13] 

Chowriappa, 

which  simulator 
correlates  with 
“gold  standard” 

Predictive:  extent  to 
which  simulator 
predicts  future 
performance 

Tergas 

[25] 

Hung  [19] 
Tergas 
[25] 

Culligan 

[28] 

Korets  [14] 

Lee  [15]  Lemer 
[26] 

[27] 

validity  classifications,  which  include  face,  content,  con¬ 
struct,  concurrent,  and  predictive  validity  [6].  Face  and 
content  validity  are  considered  subjective  approaches,  while 
the  other  three  are  objective  approaches  to  validation. 

Table  7  provides  a  summary  of  the  published  validation 
studies  for  these  simulators.  All  three  have  publications 
establishing  face,  content,  construct,  and  concurrent  vali¬ 
dation.  Only  published  studies  investigate  the  predictive 
validity  of  the  DVSS  [19,  25,  28].  Recent  presentations 
also  explore  the  validity  of  the  RoSS  curriculum  [29]  and 
the  RoSS’  HoST  procedural  modules  [30]. 

Conclusions 

Simulators  play  an  important  role  in  providing  a  training 
experience  and  a  platform  for  evaluation  of  novices  who 
are  trying  to  master  complex  skills  in  many  fields.  When  a 
task  is  simple,  consequences  for  failure  are  minimal,  and 
equipment  is  inexpensive,  there  is  little  motivation  for 


creating  a  dedicated  simulation  device.  However,  when  the 
task  to  be  mastered  is  complex,  there  is  a  need  for  a  device 
that  can  objectively  measure  the  performance  of  the  trainee 
and  provide  feedback  that  leads  to  improved  performance. 
When  the  consequences  of  a  mistake  can  be  lethal,  there  is 
a  need  for  a  safe  environment  in  which  to  develop  expertise 
without  threatening  the  wellbeing  of  others.  When  equip¬ 
ment  or  disposables  are  expensive  to  use,  there  is  a  need  for 
a  tool  that  can  provide  at  least  entry-level  familiarization 
and  skill  development  without  undue  financial  demands. 
All  three  of  these  conditions  are  characteristic  of  the  pro¬ 
cess  for  learning  robotic  surgery.  So  it  is  not  surprising  that 
market  forces  have  led  to  the  creation  of  multiple  simula¬ 
tors  of  the  robotic  system  and  the  skills  to  use  it. 

This  article  represents  the  first  part  of  a  comprehensive 
analysis  of  robotic  surgical  simulators.  The  second  part  is 
a  subjective  opinion  survey  on  the  usability  of  the  sim¬ 
ulators.  Subjects  for  this  survey  will  include  attending 
surgeons,  fellows,  residents,  and  medical  students  without 
prior  experience  using  the  simulation  devices.  The  third 
part  will  include  a  select  group  of  surgical  fellows  who 
will  participate  in  a  two-month  experiment  in  which  each 
practices  on  one  of  the  simulators,  while  their  perfor¬ 
mance  is  measured  every  2  weeks  to  assess  for  changes 
and  maintenance  of  skill  levels.  The  experiment  is 
designed  to  determine  which  simulator  has  the  greatest 
positive  impact  on  robotic  surgical  performance,  and  the 
degree  to  which  those  improvements  are  retained  across  a 
period  of  inactivity. 

The  three  simulators  described  in  this  article  are  com¬ 
plex  systems,  which  are  significantly  less  costly  than  the 
actual  da  Vinci  robotic  surgical  system  and  can  be  operated 
at  a  fraction  of  the  cost  of  the  instruments  required  by  this 
robot.  Furthermore,  da  Vinci  robots  are  predominantly  used 
for  daily  surgery,  decreasing  their  availability  for  training. 
There  are  currently  no  available  studies  directly  comparing 
the  three  simulators,  and  therefore  until  those  studies  are 
performed,  no  universal  recommendation  can  be  made  for 
one  device  over  the  other,  and  a  decision  to  use  one  sim¬ 
ulator  over  the  other  should  be  based  on  unique  and  indi¬ 
vidual  needs. 
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Mini  abstract:  This  article  exposes  the  process  of  a  national  initiative  to  develop  the  Fundamentals  of 
Robotic  Surgery  psychomotor  skills  dome,  an  assessment  tool  to  train  and  evaluate  manual  dexterity 
in  robotic  surgery.  The  tool  gathers  7  basic  tasks  into  a  single  testing  device.  The  process  introduced 
modifications  to  the  initial  design  to  account  for  ergonomics,  standardization  and  costs. 
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Abstract 

Objective:  To  expose  the  surgical  and  engineering  process  of  development  of  the  Fundamentals  of 
Robotic  Surgery  psychomotor  skills  device. 

Summary  Background  data :  Exponential  growth  of  robotic-assisted  surgery  raises  the  question  of  how 
to  assess  robotic  surgery  skills.  The  U.S  Department  of  Defense  and  governing  surgical  societies 
convened  consensus  conferences  to  develop  a  national  initiative,  resulting  in  a  curriculum:  the 
Fundamentals  of  Robotic  Surgery  (FRS),  comprised  of  an  online  curriculum  and  a  psychomotor  skills 
dome.  This  paper  describes  the  production  process  used  to  develop  the  psychomotor  skills  dome. 
Methods'.  Based  on  a  review  of  the  basic  gestures  in  robotic  surgery,  7  tasks  were  gathered  on  the 
dome  to  test  manual  dexterity:  Docking,  Ring  Tower  Transfer,  Knot  Tying,  Suturing,  4th  Arm 
Cutting,  Puzzle  Piece  Dissection,  and  Energy  Dissection. 

Results'.  The  development  began  with  a  computer-generated  design,  which  was  transformed  into  Low 
then  High  Fidelity  Prototypes.  Usability  testing  was  conducted  throughout  the  process  and 
modifications  were  made  to  the  design  of  the  device  to  account  for  ergonomics,  standardization,  and 
cost. 

Conclusions:  Final  CADs  and  specifications  were  sent  to  simulation  and  manufacturing  companies  for 
production.  Experimental  trials  are  underway  to  validate  the  effectiveness  of  the  device  in  both 
teaching  and  in  assessing  specific  skills  in  robotics. 

Keywords:  Fundamentals  of  Robotic  Surgery;  Surgical  Education;  Simulation;  Psychomotor  skills; 
Robotic  Training;  Robotic  Assessment;  Robotic  Surgery;  Psychomotor  Device;  National  Surgical 
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Introduction  and  background 

Robotic  surgery  has  been  established  as  an  innovative  approach  in  surgery  due  to  the 
development  of  a  telemanipulator  device,  which  introduced  a  new  dimension  into  surgical  tools.  The 
da  Vinci  robot  overcomes  several  laparoscopic  limitations  and  facilitates  the  performance  of 
minimally  invasive  surgery  in  complex  procedures  with  3D  vision,  7-degree-of-freedom  instrument 
movement,  hand  tremor  elimination,  motion  amplification,  and  stabilization  of  the  camera  in  many 
specialties.1'4  Thanks  to  a  computerized  interface,  the  surgeon  can  manage  the  camera  and  three 
working  arms  with  wristed  instruments  at  the  same  time.5'6  Nevertheless,  the  system  also  introduces  a 
need  for  training  to  acquire  the  unique  abilities  to  operate  the  device  and  a  certification  process  to 
ensure  a  minimal  standard  of  care  for  all  patients  undergoing  robotic  surgery. 

After  the  Halstedian  learning  model,7  training  method  has  begun  to  evolve  in  which 
simulation  is  an  essential  part  of  the  training  process  in  the  medical  field.8  This  was  first  implemented 
during  the  late  1990s  when  a  significant  increase  in  complications  was  identified  in  association  with 
the  introduction  of  minimally  invasive  laparoscopic  surgery.  Educational  material  was  created  to 
“teach  a  standard  set  of  cognitive  and  psychomotor  skills  to  practitioners  of  laparoscopic  surgery”. 
The  result  was  the  Fundamentals  of  Laparoscopic  Surgery  (FLS)  curriculum,  which  was  developed  as 
a  3 -part  program  -  online  didactic  training,  hands-on  skills  training,  and  a  high  stakes  written  and 
skills  exam-  in  laparoscopic  surgery.9  Today,  robotic  surgeons  face  a  similar  challenge  and  need  a 
specific  evaluation,  which  allows  them  to  assess  their  knowledge  and  technical  skills,  guaranteeing  a 
minimal  standard  of  care  for  all  patients.10  Some  institutions  have  attempted  to  develop  and  validate 
robotic  training  for  individual  specialties;11'15  however  the  lack  of  a  national  standard  has  pushed 
surgical  societies  (e.g.  SAGES  -Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons,  SRS 
-  Society  of  Robotic  Surgery)  to  develop  a  unified  approach  and  standard  for  robotic  skills  training. 
The  Department  of  Defense,  Veterans  Administration  and  14  surgical  specialty  societies  convened 
multiple  consensus  conferences  to  create  the  Fundamentals  of  Robotic  Surgery  (FRS).16  Participants 
included  a  total  of  more  than  80  subjects  matter  experts  involving  surgeons,  psychologists, 
psychometricians,  engineers,  simulation  experts,  and  medical  educators,  with  an  initial  working  group 
including  Richard  Satava;  Timothy  Brand;  Sanket  Chauhan;  Rafael  Coelho;  Brian  Dunkin;  Susan 
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Dunlow;  Larry  Glazerman;  Tim  Kowaleski;  Gyusung  Lee;  Ray  Leveillee;  Martin  Martino;  Sonia 
Ramamoorthy;  Bernardo  Rocco;  Daniel  Scott;  and  Rob  Sweet.17  Two  principles  guided  the 
committees’  development  process.  The  first  was  to  ensure  a  perfect  understanding  of  the  basics  of 
robotic  surgery  including  the  use  of  the  da  Vinci  robot.  The  second  was  to  design  a  psychomotor  skills 
program  focused  on  the  basic  tasks  specific  to  robotic  surgery.  In  the  first  meeting  an  international 
body  of  leaders  in  robotic  surgery  convened  to  define  cognitive,  psychomotor  and  team  training  skills 
necessary  for  surgical  competence.  The  team  focused  on  identifying  the  necessary  Outcomes 
Measures  and  Metrics  that  would  be  used  to  evaluate  robotic  surgeons.  Using  a  Delphi  model  that  was 
developed  by  the  Rand  Corporation  to  achieve  consensus  within  dynamic  and  diverse  groups  of 
participants,  the  essential  tasks  necessary  to  perform  robotic  surgery  were  identified  and  prioritized.18 
The  result  was  a  matrix  of  25  specific  robotic  surgery  concepts,  which  was  the  core  material  used  in 
the  development  and  design  of  the  FRS  Curriculum.16 

Since  the  purpose  was  to  create  a  basic  fundamental  skills  program,  particular  references  to  an 
anatomic  location  or  to  a  specific  procedure  were  avoided.  At  a  later  date,  specialties  will  develop 
their  own  independent  fundamentals  beyond  the  basic  FRS,  which  may  refer  to  specific  anatomy.  Two 
assessment  tools  were  developed:  the  first  an  online  curriculum  for  knowledge  and  team  training  skills 
and  the  second  a  set  of  physical  exercises  gathered  on  a  single  device  in  the  shape  of  a  dome  for 
psychomotor  skills  evaluation.19  The  second  part  is  the  subject  of  this  work.  The  aim  of  this  paper  is 
to  explain  the  process  utilized  to  create,  optimize  and  standardize  the  FRS  committee’s  physical 
device  based  on  their  initial  concept,  which  evolved  through  low  and  high  fidelity  prototypes  prior  to 
arriving  at  the  manufacturing  process.  This  device  will  be  used  for  the  psychomotor  skill  assessment 
in  the  FRS  Curriculum.  This  device  is  known  as  the  FRS  dome. 

Intellectual  process:  brainstorming  &  device  design 

From  the  25  concepts,  16  were  directly  linked  to  psychomotor  skills  (Table.l).  Those  skills 
reflected  the  basic  tasks  in  robotic  surgery  beginning  with  docking  (the  connection  of  the  site  cart 
patient  with  the  previous  positioned  trocars),  and  most  importantly  using  surgical  gestures  as 
dissection,  cauterization  and  suture.  The  particularity  of  the  da-Vinci  system  led  to  specific  robotic 
tasks.  The  committee  envisioned  these  tasks  summarized  in  seven  exercises  positioned  on  a  single 
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integrated  physical  device,  which  would  minimize  intervention  by  a  proctor  and  facilitate  faster 
completion  times,  on  the  surface  of  a  semi-spherical  dome.  Design  consideration  included 
compatibility  with  the  surgical  robot,  the  ability  to  address  multiple  learning  objectives,  cost 
effectiveness,  usability,  reliability  and  the  inclusion  of  validated  exercises.  A  preference  was  given  to 
tasks  that  had  existing  evidence  of  validity. 

Table.  1:  The  16  concepts  directly  linked  with  psychomotor  skills  in  robotic  surgery. 


The  seven  psychomotor  exercises  are  Docking  and  Instrument  Insertion,  Ring  Tower  Transfer, 
Knot  Tying,  Railroad  Track  Suturing,  4th  Arm  Cutting,  Puzzle  Piece  Dissection,  and  Vessel  Energy 
Dissection  (Table.2).  As  a  historical  reference,  FLS  is  composed  of  5  different  tasks:  Peg  Transfer, 
Pattern  Cutting,  Ligaturing  Loop,  Suturing  with  an  intra-corporeal  knot,  and  suturing  with  an 
extracorporeal  knot.  Some  of  the  seven  dome  tasks  are  similar  to  the  FLS  curriculum,  while  others  are 
completely  new  for  the  specific  needs  of  robotic  surgery.  Each  of  the  tasks  are  described  as  follows 
and  illustrated  in  later  figures. 

-  Docking  &  Instrument  Insertion :  Robot  docking  is  an  essential  and  unique  process  required  to 
begin  a  robotic  procedure.  Trocar  insertion  must  follow  specific  guidelines  concerning  trocar 
positioning,  attachment  to  the  robot  arms,  and  patient  cart  positioning.  These  are  all  important 
preliminary  elements  for  conducting  a  safe  surgery  and  a  mistake  at  this  phase  of  the  procedure  can 
compromise  the  entire  surgery. 

-  Tower  Transfer :  Endo wrist  manipulation  with  7-degrees  of  freedom  is  specific  to  robotic  surgery. 
This  exercise  allows  evaluation  of  these  movements.  There  is  an  analogy  between  this  exercise  and  the 
ring  transfer  exercises  created  by  the  Chamberlain  group.  But  the  FRS  tower  transfer  is  more  difficult 
because  of  the  displacement  and  rotation  of  the  target  towards  in  multiple  dimensions.20 


-  Knot  Tying :  This  is  a  common  task  performed  in  surgery.21  The  FLS  curriculum  requires  performing 
a  suture  with  intra  and  extracorporeal  knot  tying.  Both  have  been  validated  in  FLS.9  In  robotic 
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procedures  however,  only  intracorporeal  knots  are  used.  Construct  validity  for  this  exercise  in 
robotics  has  been  previously  demonstrated. 

-  Railroad  Track  Suturing :  Running  suture  is  a  basic  task  simplified  by  robotic  instruments  with  their 
7  degrees  of  freedom  movement.  The  exercise  has  been  validated  in  FLS  for  laparoscopy.9 

Both  knot  tying  and  suturing  exercises  demonstrated  construct  validity  in  robotic  surgery  using  the 
FLS  curriculum  materials,22  thus,  the  FRS  patterns  were  developed  using  different  design  with  a 
similar  concept. 

-  4th  Arm  Cutting :  The  da  Vinci  system  has  three  instrument  arms,  in  addition  to  one  camera  arm, 
which  improve  surgeon  autonomy.  Surgeons  can  only  manage  two  instrument  arms  at  once,  but  are 
able  to  switch  to  the  third  instrument  arm  and  camera  when  needed.  This  specific  robotic  task  has  been 
demonstrated  to  be  able  to  differentiate  novice  and  expert  levels.22 

-  Puzzle  Piece  Dissection :  Dissection  is  a  critical  task  performed  by  surgeons.  In  the  FLS  curriculum, 
users  dissect  a  circle  printed  on  cloth  material.  Due  to  endowrist  manipulation  and  3D  vision,  the 
dissection  is  easier  using  the  robot  than  during  a  laparoscopic  procedure.6  Therefore,  this  exercise 
utilizes  puzzle  piece,  which  is  a  more  complex  shape  that  is  more  difficult  to  accomplish. 

-  Vessel  Energy  Dissection :  This  exercise  evaluates  fine  dissection  skills  and  the  ability  to  use  bipolar 
energy  for  cauterization.  In  robotic  surgery,  the  surgeon  controls  the  energy  applied  with  the 
footswitch  pedal.  The  surgeon  has  to  conduct  a  very  precise  dissection  of  a  vessel,  which  is  embedded 
in  synthetic  fatty  tissue  before  applying  energy.  Any  damage  during  dissection  will  be  easily  identified 
by  the  release  of  colored  fluid  inside  the  simulated  vessel. 

Table  2:  Description  of  the  basic  psychomotor  skills  associated  with  the  seven  FRS  tasks. 


Exercises  Design  &  Arrangement 

The  design  group  envisioned  each  of  the  exercises  occupying  a  space  on  the  outer  surface  of  a  support 
structure,  for  which  they  selected  a  semi-spherical  dome.  They  explored  a  number  of  arrangements  of 
the  exercises  and  patterns  of  flow  from  one  station  to  the  next  on  the  surface  of  this  dome.  Different 
concepts  were  presented  as  paper  sketches  and  crude  physical  models  from  the  materials  at  hand  in  the 
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meeting  spaces.  A  “final”  sketch  that  follows  all  criteria  was  delivered  to  a  3D  digital  artist  to  create 
static  pictures  of  the  device  and  exercises,  as  well  as  animation  of  the  performance  of  each  of  these 
exercises  (Fig.l). 


Figure  1:  The  initial  3D  graphic  FRS  dome  design 
Manual  process:  makestorming 

Using  the  documentation  from  the  design  meeting  and  the  computer  generated  images  and  animations; 
the  authors  took  the  first  step  in  creating  a  physical  prototype  of  the  device.  This  physical 
brainstorming  process,  or  ’’makestorming”,  was  an  essential  step  in  determining  whether: 

•  The  single  device  could  actually  be  created. 

•  The  exercises  could  coexist  on  the  surface  of  a  dome. 

•  The  exercises  configuration  could  be  performed  with  a  surgical  robot. 

•  The  exercises  were  repeatable  with  a  high  level  of  standardization. 

•  The  outcomes  of  each  exercise  could  be  measured. 

Low  fidelity  prototypes  (LFP)  were  created  with  inexpensive  craft  materials  (Fig.2).  The  LFP  testing 
allowed  enhancement  of  the  design  in  terms  of  usability  and  organization  of  the  different  exercises. 
The  materials  were  sufficient  to  create  a  working  prototype,  which  could  be  used  to  test  fit,  function 
and  organization  as  described  above. 

Early  in  the  makestorming  process,  we  tested  processes  to  ensure  usability,  face  validity  and  content 
validity  of  the  FRS  Dome.  Surgeons  were  video  recorded  completing  all  exercises  and  the  recordings 
were  reviewed,  focusing  on  ergonomics,  material  behavior  and  appropriateness  of  the  organization  of 
the  exercises  (Fig.3).  This  allowed  rapid  trial  and  error  testing  of  the  technical  aspect,  clarifying 
requirements  and  proving  usability. 


A 

B 

C 

D 

Figure  2:  Low  Fidelity  Prototype  (LFP):  LFP1  (A);  LFP3  (B);  LFP4  (C)  &  LFP5  (D) 
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Feedback  was  solicited  from  members  of  the  design  team  around  the  globe  regarding  the  results  of 
prototyping  and  testing,  as  well  as  modifications  to  improve  the  effectiveness  of  the  design.  This 
resulted  in  dozens  of  major  and  modest  improvements  to  the  original  design  of  the  device.  Each 
change  was  tested  on  subsequent  and  increasingly  complex  prototypes.  Six  distinct  LFP  models  were 
constructed  and  tested  in  this  manner. 

The  initial  prototype  (LFP1)  was  created  using  an  8”  Styrofoam  sphere  as  the  support 
structure,  a  sheet  of  yellow  felt  material  for  the  fat  layer,  a  latex  swimming  cap  for  the  skin  layer,  straws 
for  the  embedded  vessels,  and  foam  blocks  for  the  ring  towers  (Fig.2A).  The  patterns  were  stenciled 
onto  the  surface  with  permanent  markers.  Towers  were  secured  to  the  dome  with  magnets.  The  strength 
of  the  magnets  used  to  secure  the  towers  to  the  dome  was  important  for  achieving  firm  attachment  while 
still  allowing  excessive  force  to  detach  them. 


A 


B 


Figure  3:  Early  video  captured  usability  test  results  on  the  low  fidelity  prototypes  (A:  LFP1;  B: 


LFP5). 


Multiple  design  problems  were  identified  with  the  early  iterations  of  the  LFP’s: 

•  An  8”  dome  with  one-inch  towers  was  too  large  for  the  range  of  movement  of  the  robotic 
arms.  A  smaller  dome  was  needed  to  allow  access  to  all  exercises  without  moving  the  dome. 

•  Given  the  direct  approach  necessary  for  the  exercises,  it  was  not  possible  to  visualize  or 
accurately  cut  the  large  cloverleaf  shape,  which  was  first  selected  for  the  cutting  exercise.  A 
user  could  not  reach  the  far  backside  of  the  dome  where  one  of  the  clover  leaf  petals  extended. 

•  The  weight  of  the  LFP  was  not  substantial  enough  to  prevent  the  robot  arms  from  pushing  it 
around  during  the  exercises. 

In  LFP3,  we  introduced  an  elastic  band  for  the  4th  arm-cutting  task  to  replace  the  rigid  tube  and 
allow  improved  retraction  (Fig.2B).  This  modification  worked  well  and  has  remained  nearly 
unchanged  through  the  design  phase.  We  also  worked  to  reduce  the  size  of  the  cloverleaf  shape  cutting 
pattern  in  an  attempt  to  allow  the  user  to  successfully  complete  the  360  degree  pattern  cut.  However, 
even  with  a  smaller  size,  the  users  were  unable  to  dissect  the  entire  pattern.  So,  we  began  exploring 
significantly  different  patterns,  which  were  primarily  oriented  toward  the  robotic  camera. 


10 


In  LFP4,  we  began  to  experiment  with  new  cutting  patterns,  which  could  be  accessed  from  a 
single  direct  view  of  the  dome,  which  also  created  additional  room  for  the  other  exercises  (Fig.2C). 
The  goal  of  a  new  pattern  was  to  retain  the  difficulty  of  instrument  manipulation  found  in  the 
cloverleaf  shape,  but  to  do  so  in  a  smaller  area  and  with  multiple  instances  appearing  on  the  surface. 
This  one  change  to  the  pattern  opened  the  design  to  the  possibility  of  multiple  iterative  trials  of  all  of 
the  exercises  on  the  surface.  LFP4  also  contained  multiple  versions  of  the  suturing  exercise  to 
determine  whether  it  should  be  embedded  in  the  skin  layer  or  attached  to  the  surface  as  a  separate 
item.  On  this  prototype  we  introduced  Velcro  fasteners  on  the  underside  of  the  dome  to  attach  it  to  the 
table-top  so  that  the  robot  arms  would  not  move  it  during  use. 

In  LFP5,  we  arrived  at  a  puzzle  piece  as  an  ideal  complex  shape  to  measure  challenge 
instrument  dexterity,  and  confined  the  design  to  a  smaller  area  (Fig.2D).  Based  on  the  length  of  the 
blades  of  the  da  Vinci  curved  scissor  instrument,  we  selected  an  acceptable  size  for  the  puzzle  piece 
that  allowed  the  necessary  cuts  and  turns  in  the  excising  pattern.  Prior  experimentation  with  surface 
mounted  suturing  materials  indicated  that  these  were  more  difficult  to  assess  objectively  so  the 
suturing  exercise  was  embedded  in  the  skin  layer  in  this  design. 

Then,  we  shifted  from  common  inexpensive  materials  to  custom  materials,  which  could  be 
used  in  the  final  manufacturing  of  the  device.  These  materials  were  significantly  more  expensive  than 
the  earlier  set  and  began  to  represent  the  true  texture,  appearance,  and  performance  of  the  final  device, 
imitating  skin  and  fat.  This  signaled  the  evolution  to  a  High  Fidelity  Prototype  (HFP)  7. 

The  HFP  stage  allowed  us  to  find  appropriate  materials,  to  finalize  the  organization  of  the 
exercises  for  a  better  cost-effectiveness  and  to  standardize  the  exercises.  For  each  prototype,  the  tests 
performed  were  video-recorded  for  evaluation  and  feedback  from  various  stakeholders  in  the  project 
(Fig-4). 


Figure  4:  High  Fidelity  Prototype  video-recorded  tests 
Appropriate  materials 

The  dome  structure  at  the  HFP  stage  was  created  using  a  3D  photolithographic  printer,  which 
could  create  any  custom  design  in  three  dimensions  using  plastic  and  rubber  resins.  Since  we  were  still 
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experimenting  with  the  placement  of  the  exercises,  the  first  3D  printed  dome  structure  was  created 
with  a  dimpled  golf  ball-like  surface.  This  allowed  placement  of  support  magnets  at  dozens  of 
different  locations  on  the  surface  (Fig.5A-C).  The  weight  of  this  heavier  shell  material  also  solved  the 
problem  of  anchoring  the  dome.  The  devices  weight  was  now  sufficient  so  that  the  natural  movements 
of  the  robotic  instruments  on  its  surface  did  not  displace  it. 

The  cost  of  the  CAD  drawings  and  creation  of  one  of  these  domes  was  over  $1,000  using  the 
specific  machinery,  materials,  and  software  available  in  our  facility.  With  the  flexibility  of  3D 
printing,  we  were  also  able  to  create  a  separate  top  cap  for  the  dome.  This  concept  was  part  of  the 
original  design,  but  was  difficult  to  realize  in  the  earlier  LFP's.  This  cap  provided  the  location  in  which 
the  design  committee  originally  envisioned  housing  two  of  the  exercises.  It  also  served  as  an  anchor 
device  for  the  top  edges  of  the  fat  and  skin  layers. 


A 

C 

B 

D 

Figure  5:  Initial  (A-C)  and  final  (B-D)  3-D  stereolithography  printing  shell  and  cap  of  the  dome 


For  the  fat  and  skin  layers  of  HFP,  we  purchased  silicone  materials,  which  are  used  for 
professional  medical  moulage  (i.e.  simulated  injuries  and  wounds).  After  testing  multiple  formulas  we 
found  specific  silicone  products  that  allowed  us  to  create  unique  textures  for  the  surface  or  “skin” 
layer  (Ecoflex  00-30  Platinum  Cure  Silicone)  and  the  underlying  “fatty”  layer  (Ecoflex  Gel  Platinum 
Cure  Silicone).  With  these  materials  we  could  create  layers  for  less  than  $20  per  pair,  and  were  able  to 
make  custom  blood  vessels  and  elastic  bands  for  the  respective  exercises. 

Expert  surgeons  performed  multiple  surgical  tests  to  evaluate  initial  face  and  content  validities 
of  this  new  dome.  This  included  testing  the  new  dissection  shape  with  realistic  materials,  evaluating 
the  skin  and  fat  behaviors,  and  exploring  the  best  placement  of  the  different  exercises.  We  were  able 
to  finalize  the  placement  of  the  exercises  and  select  specific  material  properties  for  the  final  product. 

Cost  effectiveness 

The  new  pattern  of  the  dissection  exercise  allowed  the  exercise  to  be  repeated  three  times  on 
the  surface  of  the  dome.  Given  this  repeating  pattern,  the  ring  tower,  knot  tying  and  suturing  exercises 
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also  could  appear  three  times  on  the  surface.  As  a  result,  a  single  set  of  skin  and  fat  layers  could  be 
used  for  up  to  three  complete  exercise  sets,  significantly  reducing  associated  costs. 

We  realized  that  the  middle  fat  layer  was  only  being  accessed  at  three  points  where  the  blood 
vessels  were  embedded  and  therefore  offered  significant  additional  space  for  dissection  exercises.  This 
allowed  us  to  add  three  more  vessels  so  users  could  perform  all  exercises  three  times  per  skin  layer 
and  six  times  per  fat  layer.  These  vessels  contained  a  red  fluid  simulating  blood  allowing  for  quick 
recognition  of  a  failure  during  the  dissection  of  the  vessel  from  the  fat  layer. 

Standardization 

As  the  dome  will  be  used  for  training  and  assessment,  a  high  level  of  standardization  is 
required.  With  the  different  tests  conduct  on  the  HFP,  we  noted  that  many  configurations  were 
possible.  This  variability  was  not  compatible  with  the  reliability  necessary  for  an  assessment  device. 
The  last  step  before  manufacturing  was  to  introduce  standardized  markers  on  the  different  pieces  of 
the  Dome.  After  settling  on  the  final  configuration  of  the  exercises,  a  new  shell  was  created  with 
divots  only  in  the  specific  location  of  the  tower  transfer  and  knot  exercises  (Fig.5B).  Navigation 
markers  were  added  on  the  dome  shell,  fat  and  skin  layers  to  ensure  proper  alignment.  A  triangular 
mark  was  added  on  the  lower  edge  of  the  dome  at  the  location  of  each  puzzle  piece.  Corresponding 
with  the  marks  were  triangular  markers  on  the  fat  and  skin  layers,  which  aligned  the  embedded  vessels 
to  the  appropriate  location  on  the  dome  shell  and  directly  beneath  the  puzzle  pieces  on  the  skin.  A 
“tongue”  was  added  to  bases  of  the  tower  exercises  that  aligned  with  markers  printed  on  the  skin  to 
standardize  the  orientation  of  each  tower.  Finally,  the  dome  cap  had  three  specific  lines,  which 
identified  alignment  for  the  towers  positioned  in  the  cap  with  (Fig.5D). 

HFP  10  presented  all  exercises  and  materials  as  they  were  expected  to  exist  in  the  final 
product.  At  this  stage,  new  surgical  tests  were  conducted  to  assess  final  usability,  ergonomics,  face 
and  content  validity.  This  version  was  used  as  the  basis  for  a  specification  document  for  constructing 
the  device.  This  captured  the  final  appearance,  purpose,  function,  and  assembly  (Fig.6  A-B)  and  was 
intended  to  allow  multiple  companies  to  create  identical  versions  of  the  device  for  sale. 
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Figure  6:  The  Computer  Aid 

ed  Design  of  the  Dome  (A-B) 

and  the  last  High  Fidelity  Prototype  (C-D) 


Using  this  final  design  for  guidance,  we  constructed  the  first  instance  of  a  completed  dome, 
“High  Fidelity  Final  1”  (Fig. 6  C-D).  This  experience  allowed  us  to  improve  the  specification 
document  so  it  could  be  followed  as  an  instruction  set  by  external  manufacturers.  The  document  and 
HFF1  were  delivered  to  a  manufacturing  company  to  begin  small  lot  production  of  the  FRS 
Psychomotor  Testing  device. 

Conclusion 

This  article  outlines  the  evolution  of  the  Dome  concept  from  the  initial  FRS  committee  expert 
brainstorming  to  the  final  prototype  for  manufacturing.  Based  on  16  objectives  recognized  as  basics 
tasks  of  robotic  surgery,  this  psychomotor  device  gathers  7  exercises  from  docking  to  suture  and 
dissection.  The  evolution  passed  through  low  fidelity  prototypes  created  with  inexpensive  materials  to 
high  fidelity  prototypes  with  resin  products,  using  different  silicone  combinations  to  create  textures 
resembling  human  tissue.  Ergonomics,  cost-effectiveness  and  standardization  were  acquired  through 
multiple  tests  during  each  stage  of  the  process. 
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Tables  and  figures  legends 

Table  1:  The  16  fundamental  tasks  directly  linked  with  psychomotor  skills  in  robotic  surgery 
Table  2:  Description  of  the  basic  psychomotor  skills  associated  with  the  seven  FRS  tasks. 

Figure  1:  The  initial  3D  graphic  FRS  dome  design 

Figure  2:  Low  Fidelity  Prototype  (LFP):  LFP1  (A);  LFP3  (B);  LFP4  (C)  &  LFP5  (D) 

Figure  3:  Early  video  captured  usability  test  results  on  the  low  fidelity  prototypes  (A:  LFP1;  B: 
LFP5). 

Figure  4:  High  Fidelity  Prototype  video-recorded  tests 

Figure  5:  Initial  (A-C)  and  final  (B-D)  3-D  stereolithography  printing  shell  and  cap  of  the  dome 
Figure  6:  The  Computer  Aided  Design  of  the  Dome  (A-B)  and  the  last  High  Fidelity  Prototype  (C-D) 
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Table  1:  The  16  fundamental  items  directly  linked  with  psychomotor  skills  in  robotic  surgery. 

Phase  Items 


Pre-operative 

-  Docking 

-  Robotic  trocar 

-  Energy  sources 

-  Clutching 

-  Camera  control 

-  Wrist  articulation 

-  Instrument  exchange 

-  Multi-arm  control 

Intra-operative 

-  Eye  hand  coordination 

-  A-traumatic  handling 

-  Dissection  -  Fine&  Blunt 

-  Cutting 

-  Needle  driving 

-  Knot  tying 

-  Suture  handling 

Post-operative  -  Undocking 
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Table  2:  Description  of  the  basic  psychomotor  skills  attached  to  the  seven  FRS  tasks. 


Exercises 

Skills 

Task  1:  Docki 

ng  &  Instrument  Inse 

m* 

rtion: 

-  Docking 

-  Instrument  insertion 

-  Eye -hand  coordination 

-  Operative  field  of  view 

Task  2:  Ring " 

rower  Transfer: 

it ^  - 

-  Eye -hand  coordination 

-  Camera  navigation 

-  Clutching 

-  Wrist  articulation 

-  A-traumatic  handling 

Task  3:  Knot  Tying: 


-  Knot  tying 


,  M 

lL 

-  Suture  handling 

-  Eye -hand  coordination 

-  Wrist  articulation 

1 

Task  4:  Railro 

ad  Track 

-  Needle  handling  &  manipulation 

-  Wrist  articulation 

-  A-traumatic  handling 

-  Eye -hand  coordination 

iiS 

Task  5: 4th  An 

n  Cutting 

-  Multiple  arm  control  &  switch 

-  Cutting 

-  A-traumatic  handling 

-  Eye -hand  coordination 

Task  6:  Puzzle  Piece  Dissection 

-  Sharp  and  blunt  dissection 

-  Cutting 

-  A-traumatic  handling 

-  Eye -hand  coordination 

-  Wrist  articulation 

Task  7:  Vessels  energy  dissection 


Energy  sources  use 
Sharp  dissection 
Cutting 

Multiple  arm  control 
A-traumatic  handling 
Eye -hand  coordination 
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Abstract: 

Purpose:  The  present  study  aims  to  evaluate  the  impact  of  different  latency  levels  on 
robotic  performances  of  surgeons  on  four  simulated  tasks. 

Materials  and  methods:  During  3  robotic  conferences,  surgeons  were  enrolled  to 
perform  4  tasks  on  a  robotic  simulator  first  without  delay  (Base)  and  then  twice  with  a 
randomly  assigned  delay  (Warm-up  and  Test)  ranging  from  100ms  to  1,000ms.  The 
simulator  provided  an  automatic  scoring  system  with  an  overall  score  based  on  several 
criteria  such  as  time,  motion,  error  metrics,  etc.  The  performances  were  collected  for 
statistical  analysis  using  the  repeated  measures  ANOVA. 

Results:  The  complete  performances  of  37  surgeons  were  collected.  In  latency  stages 
(Warm-up  and  Test),  overall  score  demonstrated  a  decreasing  trend  and  task 
completion  time  increased  gradually  as  latency  increased.  All  metrics  clearly 
deteriorated  from  non-latency  stage  (Base)  to  latency  stages  starting  at  300-500ms  for 
the  easiest  exercises  and  around  200ms  for  the  more  complex  one.  Performances 
improved  from  warm-up  to  test  in  all  exercises  and  the  improvement  was  more 
significant  in  more  complex  tasks.  The  mean  error  score  significantly  deteriorated  from 
Base  to  latency  stages  at  delays  >  500ms  (p<0.01).  Of  the  incomplete  tasks  with 
latency,  73.75  %  were  performed  under  delays  >  600  ms. 

Conclusion:  The  gradually  increasing  latency  has  a  growing  impact  on  performances, 
and  the  deterioration  starts  at  300ms.  Delays  equal  or  higher  than  700ms  are  difficult  to 
manage  especially  in  complex  tasks.  Surgeons  had  the  ability  to  adapt  to  delay  and 
may  be  trained  to  improve  their  telesurgical  performances. 
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Re:  Manuscript  reference  (JU-14-958) 

Please  find  attached  a  revised  version  of  our  manuscript  "Impact  of  delay  on  telesurgical 
performance:  Study  on  the  dV-Trainer  robotic  simulator",  which  we  would  like  to 
resubmit  for  publication  as  an  original  article  in  The  Journal  of  Urology.  We  revised  the 
paper  to  answer  all  of  the  criticisms  raised  in  the  reviewing  process.  As  you  proposed  in 
your  e-mail,  we  resubmit  the  article  as  a  new  article. 

Your  valuable  comments  and  those  of  the  reviewers  were  highly  insightful  and  enabled 
us  to  greatly  improve  the  quality  of  our  manuscript.  In  the  following  pages  are  our  point- 
by-point  responses  to  each  of  the  comments  of  the  reviewers  as  well  as  your  own 
comments. 

As  the  manuscript  is  resubmitted  as  a  new  article,  the  revisions  are  not  highlighted. 

In  accordance  with  reviewer’s  suggestions,  we  made  many  changes: 

Reviewer's  critique: 

Introduction 

End  of  second  paragraph,  it  is  quoted  that  latency  on  the  Internet  is  700-900  ms. 
From  where  are  that  data? 

We  added  a  reference. 

Methods 

Each  user  only  performed  one  latency  delay  (ie  only  at  100  ms)  or  did  they 
perform  each  delay  from  100  ms  to  1000  ms? 

Each  user  performed  within  a  single  latency.  We  did  not  have  a  user  experience  multiple 
latencies.  Completing  the  experiment  for  a  single  latency  required  a  time  of  between  30 
minutes  and  2  hours.  It  was  not  practical  to  get  attending  physicians  to  work  longer  than 
this.  Additionally,  if  a  subject  were  able  to  repeat  the  exercises  with  different  levels  of 
latency,  they  would  be  developing  new  skills  and  learning  to  improve  their  performance 
in  these  trials.  Therefore,  their  data  on  second  or  third  levels  would  not  be  comparable 
to  other  subjects  who  experienced  a  different  progression  through  multiple  latency 
levels. 

To  remove  bias,  were  users  blinded  to  the  latency  time  they  were  experiencing? 

Yes,  subjects  were  blinded  to  the  latency  they  received.  We  never  told  them  what  it  was 
for  their  trials. 


Results 

Less  than  half  of  recruited  participants  completed  their  assigned  exercises.  Can  it 
be  concluded  that  those  who  did  not  complete  did  so  because  they  were  incapable 
of  completing  (ie  from  the  severe  latency)  or  could  it  also  be  from  study  logistical 
challenges  (ie  any  flaw  or  additional  challenge  because  of  study  design)? 

Our  experience  with  all  subjects  was  that  they  exhibited  a  determination  to  complete  the 
experiment.  Depending  on  the  level  of  latency,  this  could  require  between  30  minutes 
and  2  hours  to  finish  the  experiment.  In  most  cases,  the  subjects  chose  to  terminate  an 
exercise  when  they  found  it  too  difficult  to  complete  with  their  skill  level  and  the 
assigned  latency  level.  A  few  subjects  completed  two  or  three  exercises  and  then 
returned  to  a  conference,  promising  to  return.  Those  who  did  not  return  were  likely  due 
to  the  logistics  of  the  experiment  and  the  location. 

Incomplete  data  -  since  so  much  data  were  incomplete  and  thus  not  analyzed, 
better  accounting  should  have  been  made  of  why  they  were  not  completed,  not 
just  by  authors'  conjecture. 

When  a  subject  attempted  to  complete  an  exercise  and  spent  10  minutes  or  more 
struggling,  we  are  certain  that  their  termination  of  the  study  was  due  to  a  lack  of  skill  to 
perform  at  the  given  latency  level.  Only  when  subjects  did  not  return,  as  described 
above,  do  we  believe  that  logistics  could  be  the  cause  of  the  incomplete  data. 

It  seems  difficult  to  compare  across  latency  times  when  the  authors  are  really 
comparing  different  users  across  different  latency  times.  The  study  is  not 
comparing  how  specific  set  of  users  fare  across  different  latency  times.  Rather  it 
looks  at  how  different  users  fare  across  different  latency  times.  This  is  a  flaw  in 
the  study. 

This  is  not  a  flaw  in  the  study.  As  explained  in  a  previous  question,  the  experiment  was 
designed  to  prevent  a  subject  from  learning  through  a  progression  of  different  levels  of 
latency.  Since  different  subjects  would  go  through  a  different  series  of  latency  levels, 
none  of  their  data  would  be  comparable  to  the  learning  experience  of  other  subjects.  The 
experiment  was  not  supposed  to  teach  them  to  accommodate  for  latency,  but  to  capture 
their  existing  ability  to  adapt  to  latency  with  a  short  warm-up.  We  rely  on  the 
demographic  data  and  large  numbers  of  subjects  to  identify  common  thresholds  at 
which  significant  changes  in  performance  occur. 

We  are  not  certain,  but  the  reviewers  comment  may  refer  to  the  case  in  which  we 
grouped  subjects  with  latency  levels  higher  than  700ms  for  the  purposes  of  trend 
analysis.  In  that  case,  it  is  correct  that  the  data  in  that  group  is  not  directly  comparable 
to  the  data  in  other  groups  which  contained  subjects  with  a  single  latency  level.  As 
describe  in  the  paper,  there  were  not  sufficient  subjects  at  each  of  the  higher  levels  to 
perform  an  analysis  for  each  of  700ms  through  1,000ms. 

Discussion 

I  disagree  with  excluding  "overly  complicated  exercises"  such  as  Suture  Sponge 
and  Tubes.  Arguably  they  are  the  closest  match  to  clinical  practice  of  robotic 
surgery.  Why  were  they  not  included? 

We  used  exercises  close  to  real  procedures,  such  as  Thread  the  Rings,  which  is  a 
suturing  exercise  and  Energy  Dissection.  We  avoided  exercises  like  Suture  Sponge 


because  it  is  very  long  in  time  even  without  latency  and  may  have  led  to  experimental 
sessions  or  multiple  hours  for  subjects  with  long  latencies.  We  did  use  the  Suture 
Sponge  exercise  in  another  experiment  with  medical  students  who  were  being  trained  to 
manage  delay  on  the  dV-Trainer.  We  found  this  exercise  to  be  extremely  long  for  those 
subjects  and  beyond  the  availability  of  attending  surgeons  for  the  experiment  reported 
in  this  manuscript.  Based  on  that  experience,  we  believe  it  would  have  significantly 
increased  dropout  based  on  time  constraints  in  this  study. 

We  hope  that  the  revisions  in  the  manuscript  and  our  accompanying  responses  will  be 
sufficient  to  make  our  manuscript  suitable  for  publication  in  the  Journal  of  Urology. 

We  look  forward  to  hearing  from  you  at  your  earliest  convenience. 
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Manuela  Perez,  MD,  PhD, 
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ABSTRACT: 


Purpose:  The  present  study  aims  to  evaluate  the  impact  of  different  latency  levels  on  robotic 
performances  of  surgeons  on  four  simulated  tasks. 

Materials  and  methods:  During  3  robotic  conferences,  surgeons  were  enrolled  to  perform  4 
tasks  on  a  robotic  simulator  first  without  delay  (Base)  and  then  twice  with  a  randomly 
assigned  delay  (Warm-up  and  Test)  ranging  from  100ms  to  1,000ms.  The  simulator  provided 
an  automatic  scoring  system  with  an  overall  score  based  on  several  criteria  such  as  time, 
motion,  error  metrics,  etc.  The  performances  were  collected  for  statistical  analysis  using  the 
repeated  measures  ANOVA. 

Results:  The  complete  performances  of  37  surgeons  were  collected.  In  latency  stages  (Warm¬ 
up  and  Test),  overall  score  demonstrated  a  decreasing  trend  and  task  completion  time 
increased  gradually  as  latency  increased.  All  metrics  clearly  deteriorated  from  non-latency 
stage  (Base)  to  latency  stages  starting  at  300-500ms  for  the  easiest  exercises  and  around 
200ms  for  the  more  complex  one.  Performances  improved  from  warm-up  to  test  in  all 
exercises  and  the  improvement  was  more  significant  in  more  complex  tasks.  The  mean  error 
score  significantly  deteriorated  from  Base  to  latency  stages  at  delays  >  500ms  (p<0.01).  Of 
the  incomplete  tasks  with  latency,  73.75  %  were  performed  under  delays  >  600  ms. 
Conclusion:  The  gradually  increasing  latency  has  a  growing  impact  on  performances,  and  the 
deterioration  starts  at  300ms.  Delays  equal  or  higher  than  700ms  are  difficult  to  manage 
especially  in  complex  tasks.  Surgeons  had  the  ability  to  adapt  to  delay  and  may  be  trained  to 
improve  their  telesurgical  performances. 
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INTRODUCTION 


Robotic  surgery  was  noted  to  be  in  its  infancy  in  2004, 1  but  now  this  advanced 
technology  is  on  its  way  to  young  adulthood  and  has  even  become  a  standard  in  complex 
surgery.  The  mature  experience  will  likely  include  the  achievement  of  remote  telesurgery, 
the  future  challenge  for  robotic  surgeons.4,5 

The  first  transatlantic  human  telesurgery  procedure  was  performed  in  2001. 6  Since  the 
proof  of  concept,  telesurgery  remains  a  complex  and  uncommon  process  that  holds  promise  in 
overcoming  challenging  situations  (remote  medicine  for  underserved  regions,  surgery  in  the 
battlefield,  surgery  in  space,  etc.).  ’  Many  teams  have  worked  on  the  telesurgery  process  and 
try  to  achieve  remote  telesurgery  procedure  using  available  technical  resources  for  the  video 
flux  transfer.7,9,10  In  telesurgery,  the  control  signal  sent  from  the  master  console  is  transferred 
over  a  network  to  the  robot  arm  followed  by  a  corresponding  movement  of  the  surgical 
instruments.  The  laparoscopic  videos  are  then  returned  to  the  surgeon  site.  The  information 
transmission  requires  an  encoding,  transmitting,  and  decoding  process  in  which  a  time  delay 
is  inevitably  produced.  Latency  is  correlated  with  the  amount  of  data  transmitted  and  the 
quality  of  network.  The  first  transatlantic  human  robotic  remote  procedure  used  sophisticated 
dedicated  Asynchronous  Transfer  Mode  (ATM)  lines  with  a  delay  of  transmission  around 
150ms.6  Dedicated  lines  however  are  not  always  feasible  in  clinical  routine.  Given  the  cost 
effectiveness,  an  easily  accessible  resource  such  as  the  Internet  would  be  a  good  and 
affordable  choice,  but  the  availability  of  the  network  would  be  at  the  price  of  a  significant 
latency,  measuring  approximately  700-900ms  depending  on  the  speed  of  the  network  and  the 
distance  between  to  the  two  points.11 

This  latency  may  impact  the  surgical  performances,  and  extent  to  which  this  happens 
must  be  clarified  before  the  further  implementation  of  telesurgery.  Two  levels  need  to  be 
determined:  first  is  the  smallest  latency  detected  by  surgeons  that  will  influence  the 
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performance  and  second  is  the  latency  that  makes  the  surgery  unsafe,  which  means  a  delay 
associated  with  an  increase  in  errors.  A  previous  study  on  this  topic  highlighted  the  impact  of 
delay  on  performance  degradation  using  the  dV-Trainer®.  The  authors  evaluated  the  effects  of 
delay  varying  between  100-1,000  ms  and  found  that  latencies  <  300  ms  had  a  small  impact  on 
performance.  Subjective  evaluation  then  suggested  that  surgery  became  quite  difficult  at 

delays  >  700ms.  In  this  study,  subjects  were  medical  students  trained  to  perform  delayed 
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exercises  on  the  simulator. 

The  present  study  aims  to  evaluate,  on  a  large  population  of  surgeons,  the  impact  of 
different  latency  levels  on  performances  in  four  simulated  robotic  tasks. 


MATERIAL  AND  METHODS 

Exercises: 

We  designed  a  prospective,  randomized,  observational  study  conducted  on  the  robotic 
surgical  simulator  dV-Trainer®  (Mimic  technologies  Inc,  Seattle,  USA).  This  tool  has 
demonstrated  face,  content,  construct  and  concurrent  validity  in  previous  studies.13,14  Based 
on  expert  opinion,  we  chose  4  exercises  (Figured)  for  the  test  that  would  be  performed  in  a 
constant  order  from  the  easiest  to  the  most  complex  -  Peg  Board  1  (PB1):  pick  up  and  transfer 
rings  sequentially  from  the  peg  board  to  a  single  peg  on  the  floor;  Camera  Targeting  2  (CT2): 
manipulate  the  camera  to  precisely  focus  and  zoom  on  a  target  sphere;  pick  up  and  move  a 
stone  into  a  designated  baskets;  Thread  the  Ring  1  (TR1):  pass  a  needle  and  suture  through  a 
number  of  flexible  eyelets;  and  Energy  Dissection  1  (EDI):  isolate  a  large  blood  vessel  by 
cauterizing  and  cutting  small  branching  blood  vessels  that  anchor  the  large  vessel.  Both  basic 
(endowrist  manipulation,  camera  control  and  clutching)  and  challenging  (suturing,  dissection) 
skills  were  covered  with  those  exercises.  We  introduced  fixed  latencies  into  the  programming 
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system  of  the  dV-Trainer  between  the  gesture  on  the  grips  and  the  visual  feedback  on  the 
console. 

After  Institutional  Review  Board  approval,  we  recruited  volunteers  -  fellow  and  attending 
surgeons-  during  3  Robotic  Conferences  in  2012.  Each  subject  received  a  unique 
Identification  Number  under  which  all  data  was  collected  and  then  completed  a  questionnaire 
concerning  demographic  data,  surgical  experience  and  related  activities.  The  delays  ranged 
from  100  to  1,000  milliseconds  (ms)  with  increments  of  100  ms.  Each  participant  received  a 
unique  randomly  assigned,  delay  under  which  they  would  perform  the  experiment  on  the  dV- 
Trainer.  They  performed  the  delayed  trials  with  a  single  latency  and  were  blinded  to  the 
latency  they  received. 

Procedure: 

Before  the  trial  on  the  simulator,  each  participant  received  a  standardized  short  instruction  on 
the  dV-Trainer’ s  usability  defined  as  an  acquaintance  period.  After  that,  they  performed  the  4 
exercises  in  order  with  no  delay.  The  results  provided  their  baseline  performance  (Base). 
Subjects  then  performed  the  first  delayed  trial  with  the  randomly  assigned  latency  (Warm-up) 
and  finally  repeated  the  4  exercises  once  again  with  the  same  delay  (Test).  The  Warm-up 
period  allowed  them  to  get  familiar  with  latency  and  to  acquire  short-term  adaptation. 

Metrics: 

The  dV-trainer  involves  a  built-in  scoring  system.  Values  and  scores  of  the  following  metrics 
were  automatically  recorded  after  each  attempt:  time  to  complete  the  exercise  (seconds), 
instrument  motion  (centimeters),  master  workspace  range  (centimeters),  excessive  instrument 
force  (seconds),  instruments  out  of  view  (centimeters),  instrument  collisions  and  drops. 

A  proficiency-based  overall  score  was  then  given  based  on  the  results  of  above  metrics  for 
each  exercise.  We  also  calculated  the  mean  error  score  of  the  error  metrics  in  each  exercise. 
The  error  metrics  include  drops  (in  all  the  exercises  but  EDI),  instrument  collisions,  excessive 
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instrument  force,  instruments  out  of  view,  as  well  as  blood  loss  and  misapplied  energy  time 
(specific  in  EDI).  The  higher  the  mean  score  is,  the  lower  the  error  rate. 

Statistical  methods: 

All  data  was  analyzed  using  R  Statistical  Software.  Performances  across  exercises  within  the 
three  periods  (Base,  Warm-up,  and  Test)  were  compared  using  the  repeated  measures 
ANOVA  (mixed-effects  model).  Statistical  significance  was  determined  at  p  <  0.05 


RESULTS 

A  total  of  63  subjects  enrolled  in  the  study  and  were  able  to  complete  the  baseline  and 
delayed  levels.  Twenty-six  were  unable  to  complete  the  delayed  trials  (41.2  %).  Some 
participants  did  not  perform  or  finish  all  the  exercises. 

Complete  data 

Final  data  was  derived  from  37  subjects  (30  attendings,  7  fellows)  who  completed  exercises 
including  both  the  base  and  the  delay.  Twenty-three  had  robotic  experience,  with  an  average 
of  2.7  years  and  a  range  of  (1-9)  years.  The  number  of  subjects  at  each  latency  level  is 
presented  in  Table  1. 

Overall  tendency  by  exercise 

Overall  score  in  all  latency  stages  (Warm-up  and  Test)  demonstrated  a  decreasing  trend  as 
latency  increased.  Meanwhile,  task  completion  time  in  latency  stages  increased  gradually  as 
the  latency  levels  increased.  A  linear  regression  was  fit  to  the  task  completion  time  in  the  Test 
stage  of  the  four  exercises  (p  <  0.05).  Instrument  collisions  of  PB1  in  the  Test  stage 
demonstrated  a  similar  increasing  trend  and  was  also  fit  to  a  linear  regression  {p  =  0.0378). 
Overall  score,  task  completion  time,  instrument  motion,  and  errors  clearly  deteriorated  from 
the  non-latency  stage  (Base)  to  latency  stages  at  300  ms  and  above  in  PB1  (Figure  2).  This 
deterioration  of  the  performance  was  evident  at  500  ms,  100  ms,  and  200  ms  in  CT2  (Figure 
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3),  TR1  (Figure  4),  and  EDI  (Figure  5),  respectively.  Performances  improved  from  Warm-up 
to  Test  in  all  of  the  four  exercises  and  the  improvement  was  more  significant  in  the  complex 
exercises  (EDI  and  TR1),  than  in  the  basic  exercises  (PB1  and  CT2). 

Comparisons  by  latency  group 

The  next  analysis  was  comparing  the  performance  of  each  exercise  across  latency  levels  using 
repeated  measures.  The  comparisons  between  the  three  stages  (Base,  Warm-up,  Test)  were 
performed  in  task  completion  time  (Figure  6),  instrument  motion  (Figure  7),  and  mean  error 
score  (Figure  8).  At  600  ms,  only  one  person  completed  the  TR1  exercise.  Data  of  this 
exercise  were  thus  not  used  for  statistical  analysis.  At  delays  >  700  ms,  the  subject  number 
was  limited  at  each  latency  level;  the  data  was  thus  merged  together  for  statistics  (group  700- 
1,000  ms).  Comparisons  were  performed  with  conglomerate  data  from  both  PB1  and  CT2. 

Task  completion  time  was  shorter  in  Test  versus  Base  and  Warm-up  at  100  ms  (p  < 
0.05).  It  clearly  deteriorated  from  Base  to  the  two  latency  stages  (Warm  up  and  Test)  at  delays 
higher  than  300  ms,  and  the  statistical  significance  was  achieved  at  delays  >  400  ms  (p  < 
0.01). 

Instrument  motion  significantly  increased  from  Base  to  the  two  latency  stages  at  400 
ms,  500  ms,  and  600  ms  (p  <  0.05).  At  700-1,000  ms,  instrument  motion  deteriorated  from 
Base  to  Warm-up  and  then  improved  from  Warm-up  to  Test  (p  <  0.05).  There  was  no 
significant  difference  between  Base  and  Test. 

The  mean  error  score  significantly  deteriorated  from  Base  to  latency  phases  at  delay  > 
500  ms  (p  <  0.05). 

Incomplete  data 

Eighty  incomplete  exercises  in  latency  stages  derived  from  26  subjects  were  identified.  They 
included  18  PB1,  18  CT2,  26  TR1  and  18  EDI  (Table  2).  Subjects  were  physically  unable  to 
complete  these  delayed  exercises.  Fifty-nine  (73.75  %)  tasks  were  not  completed  for  delays  > 
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600  ms,  and  53  (66.25  %)  were  stopped  by  the  subject  at  a  mean  time  of  9.8  min  (586.01  ± 
14.54  sec)  (Figure  9). 

For  Peg  Board,  mean  drops  for  uncompleted  delayed  trials  was  4.4  times  higher  than  for  un¬ 
delayed  exercises,  mean  collision  was  3.4  times  higher,  mean  excessive  force  was  12  times 
higher,  and  mean  instrument  out  of  view  was  more  than  10  times  higher. 

For  Camera  Targeting  2,  the  uncompleted  delayed  trials  had  a  mean  excessive  instrument 
force  that  was  2.6  times  higher  than  for  un-delayed  exercises  and  a  mean  instrument  out  of 
view  which  was  3.8  times  higher.  For  Thread  the  Ring,  for  delayed  uncompleted  trials  the 
mean  for  instrument  collisions  was  2.7  times  higher  than  for  un-delayed  exercises  and  mean 
instrument  out  of  view  was  19  times  higher.  For  Energy  Dissection,  the  mean  blood  loss  for 
uncompleted  delayed  exercises  was  more  than  3  times  higher  than  for  non-delayed  exercises 
(Table  3). 


DISCUSSION 

We  aimed  to  determine  the  latency  effects  on  surgical  performances  in  real  surgeons  to 
establish  the  acceptable  delays  in  telesurgery.  Overall,  the  gradually  increasing  latency  has  an 
increasing  impact  on  performances,  and  significant  performance  deterioration  started  at 
300ms.  Latencies  of  100  and  200  ms  seemed  to  have  no  clear  effect,  and  the  100  ms  group 
had  improving  performance  from  the  Base  to  the  Test  stage.  This  improvement  likely 
corresponds  to  the  learning  effects  of  basic  simulator  manipulation  and  further  proves  that 
100  ms  is  of  no  significant  influence.  For  the  superior  threshold,  delays  equal  or  higher  than 
700  ms  seem  to  be  difficult  to  manage  especially  in  complex  tasks.  Only  one  subject  was  able 
to  complete  the  most  complex  exercises  at  700  ms  and  for  delays  from  800  to  1000  ms,  only 
the  easiest  exercises  (PB1,  CT2)  were  finished.  In  the  previous  study  with  medical  students, 
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the  same  threshold  was  highlighted  and  the  authors  suggested  telementoring  as  a  safer 
choice.  Telementoring  is  an  application  of  telemedicine  that  involves  the  remote  guidance  of 
a  procedure  when  the  operator  has  limited  experience  with  the  technique.15  However,  in  this 
study,  the  error  rate  significantly  increased  from  non-latency  to  latency  stage  at  delays  >500 
ms,  which  may  indicate  an  increase  of  surgical  risk.  We  would  consider  this  value  as  the 
superior  threshold  and  telesurgery  should  not  be  recommended  in  this  condition  for  delay 
untrained  surgeons.16  This  does  not  mean  that  procedures  have  not  nor  cannot  be  performed  at 
higher  latency  levels,  however  research  is  limited  and  lacks  outcome  data.  In  a  previous 
published  study,  a  nephrectomy  was  performed  on  a  swine  under  a  delay  of  900  ms.  Two 
surgeons  performed  the  procedure,  one  in  the  remote  site  console  and  one  in  the  local  site 
console.11  In  this  article,  no  objective  data  was  provided  including  time  to  complete  the 
procedure,  error  metrics  or  stress  evaluation. 

1  n 

Surgeons  have  been  shown  to  have  the  ability  to  adapt  to  delays.  In  our  study, 
performances  improved  from  Warm-up  to  Test  in  all  the  four  exercises.  Training  to  the 
specific  latency  was  achieved  in  all  four  exercises  due  to  the  ordering  of  the  tasks.  For 
example,  subjects  performed  one  attempt  of  EDI  with  a  certain  delay  in  Warm-up  and  then 
completed  one  by  one  PB1,  CT2  and  TR1  in  test  stage  before  the  final  attempt  of  EDI.  This 
further  confirms  that  surgeons  could  be  trained  on  delay  to  improve  their  telesurgical 
performances. 

Our  results  also  demonstrate  that  the  impact  of  latency  depend  on  the  difficulty  of 
procedures.  Latency  affected  performances  on  different  levels  for  the  four  chosen  exercises; 
the  performance  deterioration  started  at  a  high  delay  (500  ms)  for  the  simple  exercise  CT2 
and  at  a  low  one  (100  ms)  in  the  more  challenging  TR1.  This  fact  indicates  that  the  minimum 
influential  and  the  maximum  acceptable  delays  could  be  different  in  surgical  procedures  with 
different  complexity. 
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We  chose  challenging  exercises  with  features  similar  to  real  surgical  procedures,  as 
with  Thread  the  Rings  and  Energy  Dissection;  but  avoided  overly  complicated  and  long 
exercises  like  “Suture  Sponge”  or  “Tubes”  considering  that  many  surgeons  were  not 
sufficiently  familiar  (or  proficient)  with  the  robot  and  the  simulator.  In  this  study,  few  tasks 
were  completed  at  delays,  higher  than  700  ms,  so  it  is  anticipated  that  the  results  would  be 
even  worse  if  applying  more  challenging  exercises. 

Participants  appeared  to  be  focused  on  task  completion.  The  mean  duration  of  attempt 
was  7.5  min.  per  each  exercise.  Even  in  the  identified  80  incomplete  exercises  only  a  few 
subjects  stopped  shortly  after  beginning.  This  suggests  that  surgeons  were  interested  in  the 
telesurgical  procedure  and  placed  a  reasonable  effort  into  completing  it,  thereby  minimizing 
bias.  When  subjects  were  unable  to  complete  delayed  exercises,  the  most  clearly  impacted 
elements  in  terms  of  error  varied  according  to  the  exercise.  Drops,  collisions,  excessive  force 
and  instrument  out  of  view  are  most  frequent  in  Peg  Board  and  Camera  Targeting  due  to  the 
high  level  of  instrument  motion.  The  ring  transfer  between  graspers  explained  the  increase  of 
drops  and  collisions  in  Peg  Board  and  Thread  the  Rings.  The  ability  to  control  bleeding  was 
sharply  diminished  under  higher  levels  of  delay  in  Energy  Dissection. 

This  study  has  potential  limitations,  even  though  we  recruited  more  than  sixty 
surgeons,  and  the  final  completion  rate  was  lower  than  expected.  Moreover,  many  surgeons 
failed  to  complete  the  process  for  tasks  at  high  delays  due  to  the  difficulty  of  controlling  the 
instruments.  In  addition,  all  subjects  were  novices  in  telesurgery  and  the  effects  of  delay  on 
their  movements  since  this  technology  is  currently  only  available  in  research  settings. 

A  complementary  study  will  be  necessary  to  assess  the  performance  degradation 
induced  by  latency  on  robotic  surgery  experts,  and  to  identify  the  telesurgical  training  that 
could  be  used  to  overcome  the  challenges  associated  with  this  environment. 
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CONCLUSION 


Surgical  performances  deteriorate  gradually  as  latency  increases.  The  impact  of  delay  depends 
on  the  difficulty  of  the  procedure.  Overall,  delays  of  100-200  ms  have  no  significant  impact, 
and  >  500  ms  cause  an  increase  in  surgical  risk  for  untrained  surgeons.  Surgery  becomes 
extremely  difficult  at  delays  higher  than  700  ms  and  should  be  avoided  except  for 
telementoring.  Surgeons  have  the  ability  to  adapt  to  latency  and  they  may  be  trained  to 
improve  their  telesurgical  performances. 


Key  of  Definitions  for  Abbreviations 

ATM  line:  Asynchronous  Transfer  Mode  line 

ms:  millisecond 

PB1:  Peg  Board  1 

CT2:  Camera  Targeting  2 

TR1:  Thread  the  Ring  1 

EDI:  Energy  Dissection  1 
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Figure  and  table  legends: 

Table  1:  Numbers  of  subjects  in  each  exercise  at  different  latency  levels  (ms=  millisecond;  *= 
data  not  used  for  statistical  analysis) 

Table  2:  Numbers  of  incomplete  tasks  at  each  latency  level  (ms=  millisecond) 

Figure  1:  DV-Trainer®  exercises:  Peg  Board  (A);  Camera  targeting  (B);  Thread  the  Ring  (C) 
and  Energy  Dissection  (D) 

Figure  2:  Change  of  metrics  with  latency  in  Peg  Board  1:  overall  score  (A),  task  completion 
time  (B),  instrument  motion  (C),  instrument  collisions  (D) 

Figure  3:  Change  of  metrics  with  latency  in  Camera  Targeting  2:  overall  score  (A),  task 
completion  time  (B),  instrument  motion  (C),  instruments  out  of  view  (D) 

Figure  4:  Change  of  metrics  with  latency  in  Thread  the  Rings  1:  overall  score  (A),  task 
completion  time  (B),  instrument  motion  (C),  instrument  collisions  (D) 

Figure  5:  Change  of  metrics  with  latency  in  Energy  Dissection  1:  overall  score  (A),  task 
completion  time  (B),  instrument  motion  (C),  misapplied  energy  time  (D) 

Figure  6:  Comparisons  of  task  completion  time  cross  exercise  at  different  latency  levels  ( *p  < 
0.05;  **  p  <  0.01;  ***  p  <  0.001) 

Figure  7:  Comparisons  of  instrument  motion  cross  exercise  at  different  latency  levels  (*p  < 
0.05) 

Figure  8:  Comparisons  of  mean  errors  score  cross  exercise  at  different  latency  levels  ( *p  < 
0.05;  **  p<  0.01) 

Figure  9:  End  times  of  the  incomplete  exercises  in  latency  stages  at  each  latency  level. 
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Key  of  Definitions  for  Abbreviations  (only  include  abbreviations  used  3  times  or  more  in  manuscript) 


Key  of  Definitions  for  Abbreviations 


ATM  line:  Asynchronous  Transfer  Mode  line 

ms:  millisecond 

PB1:  Peg  Board  1 

CT2:  Camera  Targeting  2 

TR1 :  Thread  the  Ring  1 

EDI:  Energy  Dissection  1 


Table  1 


Table  1:  Numbers  of  subjects  in  each  exercise  at  different  latency  levels  (ms=  millisecond)(*data  not 
used  for  statistical  analysis) 


Delay  (ms.) 

100 

200 

300 

400 

500 

600 

700 

800 

900 

1000 

Peg  Boardl 

4 

7 

2 

5 

4 

6 

1 

1 

- 

1 

Camera  Targeting2 

3 

6 

2 

5 

4 

4 

1 

1 

2 

1 

Thread  the  Ringl 

3 

7 

2 

6 

4 

1* 

1* 

- 

- 

- 

Energy  Dissectionl 

4 

7 

2 

7 

3 

3 

1* 

- 

- 

- 

Table  2 


Table  2:  Numbers  of  incomplete  tasks  at  each  latency  level  (ms=  millisecond) 


Delay  (ms) 

100 

200 

300 

400 

500 

600 

700 

800 

900 

1000 

Peg  Board  1 

1 

1 

2 

- 

1 

1 

2 

2 

7 

1 

Camera  Targerting2 

- 

2 

1 

2 

2 

2 

1 

2 

5 

1 

Thread  the  Ringl 

1 

1 

1 

1 

1 

5 

2 

4 

7 

3 

Energy  Dissection  1 

- 

1 

1 

- 

2 

2 

1 

2 

6 

3 

Total  (80) 

2 

5 

5 

3 

6 

10 

6 

10 

25 

8 

Table  3 


Table  3:  Incomplete  exercise  error  data  compared  to  un-delayed  exercises  for  the  same 
subjects 


Exercise 

Drops 

Excessive 

Collision 

Instrument  out 

Blood  lose 

Misapplied 

Delay 

(mean) 

force  (s) 
(mean) 

(mean) 

of  view  (mm) 
(mean) 

(ml) 

(mean) 

energy  (s) 
(mean) 

Peg  Board 

No  delay 

0.5 

1.4 

3.7* 

2.6 

Delayed- 

Uncompleted 

2.2 

17.23 

12.9 

28.9 

Camera 

No  delay 

0.  33 

33.2 

2.8 

45.5 

Targeting 

Delayed 

Uncompleted 

0.25 

88.2 

3.1 

173.25 

Thread  the 

No  delay 

0.4 

0.25* 

4.8 

1.1 

Ring 

Delayed 

Uncompleted 

0.65 

4.6 

13.07 

21.8 

Energy 

No  delay 

3.8* 

1.7 

138.1 

36.5 

Dissection 

Delayed 

Uncompleted 

26.1 

5.5 

456.8 

41.5 

*One  subjects  deleted  in  un-delayed  exercise  because  of  incongruent  data 
Peg  Board:  one  subject  with  45  collisions 
Thread  the  Ring:  one  subject  with  76  excessive  force 
Energy  Dissection:  one  subject  with  229.4  excessive  force 
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INTRODUCTION 

Robotic  surgery  has  introduced  a  new  dimension  into  the  surgical  field.  With  the  introduction  of  robotic  technology 
between  patient  and  surgeon,  a  need  to  master  new  skills  has  emerged.  Medicine  has  come  to  the  conclusion  that  the 
Halstedian  training  model  (See  one,  do  one,  teach  one)  is  no  longer  sufficient  for  teaching  complex  skills,  especially 
robotic  surgical  skills  (Cameron,  1997).  A  number  of  simulators  have  been  developed  to  support  training  and  skill 
assessment  in  robotic  surgery.  The  currently  available  dedicated  robotic  simulators  include:  the  da  Vinci  Skills 
Simulator  (dVSS)  by  Intuitive  Surgical  Inc.,  also  known  as  the  “Backpack  Simulator”;  the  dV-Trainer  from  Mimic 
Technologies  Inc.;  and  the  RoSS  by  Simulated  Surgical  Sciences  LLC  (Figure  1).  The  purpose  of  these  simulators  is 
to  train  surgeons  prior  to  using  the  actual  system  and  to  allow  them  to  acquire  the  necessary  robotic  skills  to  perform 
a  safe  surgery.  All  of  these  da  Vinci  simulators  utilize  a  visual  scene  that  is  presented  in  a  computer  generated  3D 
environment  providing  challenging  tests  for  practicing  dexterity  and  machine  operations.  Originally,  the  simulated 
exercises  trained  basic  robotic  skills;  however  with  advances  in  technology,  surgeons  can  now  train  for  specific 
procedures  (e.g.  nephrectomy  and  hysterectomy). 


Figure  1.  Simulators  of  the  da  Vinci  robotic  surgical  system 


Our  hospital  research  laboratory  has  purchased  each  of  these  three  simulators  for  the  purpose  of  studying  their 
effectiveness  and  applying  them  to  the  education  of  robotic  surgeons,  specifically  for  the  Department  of  Defense 
(DoD).  The  DoD  is  interested  in  the  effectiveness  of  the  simulators  to  train  military  surgeons  prior  to  and  after 
returning  home  from  deployments.  This  research  is  structured  as  three  distinct  stages. 


From  the  first  stage  of  this  work,  the  authors  summarized  the  objective  characteristics  of  the  three  systems.  This 
included  descriptions  of  the  exercises  offered  in  each,  metrics  used  to  evaluate  students,  overview  of  the  system 
administration  functions,  physical  dimensions  and  configurations  of  the  equipment,  and  comparisons  of  the  costs  of 
the  devices  and  their  support  equipment  (Smith  &  Truong,  2013).  In  the  first  simulator,  the  trainee  sits  at  and 
operates  the  simulated  environment  using  the  actual  da  Vinci  surgical  console.  The  simulator  is  a  custom  computer 
appended  to  the  surgical  console  through  the  actual  surgical  data  port.  While  the  simulator  costs  approximately 
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$100,000,  the  surgical  console  costs  $500,000  incurring  an  investment  of  $600,000.  Using  this  simulator,  users  can 
train  using  the  actual  hardware  they  would  use  during  surgery;  however,  this  requires  the  use  of  the  surgical  console 
that  may  be  needed  to  conduct  surgeries.  Most  hospitals  may  not  have  a  dedicated  training  console,  meaning  that 
users  would  not  have  appropriate  access  to  the  simulator.  The  second  is  a  standalone  system  that  utilizes  a 
graphic/gaming  computer,  connected  to  a  custom  desktop  viewing  and  control  device  that  replicates  the  hardware  of 
the  da  Vinci  surgeon’s  console.  This  system  shares  similar  software  with  the  dVSS,  but  does  not  require  the  use  of 
any  actual  da  Vinci  hardware.  The  cost  of  this  simulator  is  approximately  $100,000.  The  third  is  composed  of  a 
completely  customized  replica  of  the  da  Vinci  surgeon’s  console.  Internally  the  simulator  contains  a  graphic 
computer,  a  3D  monitor,  and  commercial  Omni  Phantom  haptic  controllers.  This  simulator  uses  unique  software  and 
is  a  little  more  than  $100,000  (Smith  &Truong,  2013). 

This  paper  reports  on  the  second  stage  of  this  research,  in  which  the  validity  and  usability  of  the  simulators  is 
examined.  The  third  stage  will  be  a  measure  of  learning  effectiveness  using  the  systems. 

Validity  in  Surgical  Simulation 

The  validity  of  medical  and  surgical  simulators  is  usually  measured  by  the  categories  defined  by  McDougal  (2007). 
This  paper  defines  the  most  commonly  recognized  forms  of  validation  as:  face,  content,  construct,  concurrent,  and 
predictive  validity.  Face  validity  is  typically  assessed  informally  by  users  and  is  used  to  determine  whether  the 
simulator  is  an  accurate  representation  of  the  actual  system  (i.e.  the  realism  of  the  simulator).  Content  validity  is  the 
measure  of  the  appropriateness  of  the  system  as  a  teaching  modality.  Experts  who  are  knowledgeable  about  the 
device  typically  assess  this  via  a  formal  evaluation.  Construct  validity  is  the  ability  of  a  simulator  to  differentiate 
between  the  performances  of  experienced  users  and  those  who  are  novices.  Concurrent  validity  is  the  extent  to 
which  the  simulator  correlates  with  the  “gold  standard”  and  predictive  validity  is  the  extent  to  which  the  simulator 
can  predict  a  user’s  future  performance.  Collectively,  concurrent  and  predictive  validity  are  known  as  criterion 
validity  and  are  used  as  measures  of  the  simulator’s  ability  to  correlate  trainee  performance  with  their  real  life 
performance.  Face  and  content  validity  are  most  effective  in  evaluating  the  ability  of  a  simulator  to  train  a  surgeon; 
however  construct,  concurrent,  and  predictive  validity  are  most  useful  for  evaluating  the  effectiveness  of  a  simulator 
to  assess  a  trainee. 

The  validity  of  all  three  simulators  has  been  tested  and  reported  separately  for  the  da  Vinci  skill  simulator  (Hung, 
Zehnder,  Patil,  2011;  Kelly,  Margules,  Kundavaram,  2012;  Liss,  Abdelshehid,  Quach,  2012),  the  dV-Trainer 
(Kenney,  Wszolek,  Gould,  Libertino,  Moinzadeh,  2009;  Sethi,  Peine,  Mohammadi,  2009;  Lee,  Mucksavage,  Kerbl, 
2012)  and  the  RoSS  (Seixas-Mikelus,  Kesavadas,  Srimathveeravalli,  2010;  Stegemann  et  al.,  2013;  Colaco,  Balica, 
Su,  2012;  Raza  et  al.,  2013).  To  our  knowledge  only  one  publication  has  compared  features  of  two  of  the  simulators, 
but  no  comparative  studies  have  been  performed  with  all  three  of  the  systems  (Liss  MA,  Abdelshehid  C,  Quach  S., 
2012).  Thus,  the  current  study  aimed  to  compare  all  three  commercially  available  da  Vinci  simulators  and  detail  the 
findings  for  face,  content,  and  construct  validity  for  the  three  systems. 


METHODS 

Recruitment 

Participants  in  this  study  included  medical  students,  residents,  fellows,  and  attending  physicians.  Participants  were 
recruited  from  the  University  of  Central  Florida  Medical  School,  courses  held  at  the  Nicholson  Center,  and  two 
medical  robotic  conferences  (World  Robotics  Gynecology  Congress  and  Society  of  Robotic  Surgeons  Scientific 
Meeting).  Subjects  were  excluded  from  participating  if  they  indicated  that  they  had  participated  in  a  formal  robotic 
simulation- training  course. 

Each  participant  was  categorized  into  one  of  three  groups  (i.e.  Expert,  Intermediate,  or  Novice)  according  to  the  self- 
reported  number  of  robotic  cases  (i.e.  procedures)  he  or  she  had  performed.  Individuals  performing  0-19  robotic 
cases  in  which  they  had  50%  or  greater  console  time  were  categorized  as  Novices,  individuals  with  20-99  robotic 
cases  were  considered  to  be  Intermediates,  and  individuals  with  100  or  more  cases  were  considered  to  be  Experts. 

Materials 


2014  Paper  No.  14168  Page  4  of  11 


Inter  service/industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


After  being  categorized  into  an  experience  level,  each  participant  was  assigned  a  specific  order  in  which  they  used 
each  of  the  simulators  (Figure  2).  This  order  system  was  used  to  identify  and  potentially  eliminate  any  bias  that  may 
exist  by  using  a  specific  system  first.  All  participants  completed  one  exercise  on  each  of  the  simulators.  The  tasks 
chosen  were  Peg  Board  1  in  both  the  dV-Trainer  and  the  dVSS  and  Ball  Placement  1  in  the  RoSS.  The  same  task 
was  used  for  both  the  dV-Trainer  and  the  dVSS  because  these  systems  share  similar  software  and  exercises.  The 
RoSS  software  contains  unique  exercises  and  Ball  Placement  1  is  designed  to  teach  the  same  skills  as  Peg  Board  1. 


Figure  2.  Rotating  order  of  use  by  subjects,  with  survey  order. 


After  each  exercise  on  each  simulator,  participants  completed  a  post  questionnaire  (Survey  1),  which  asked  for 
feedback  regarding  their  experience  on  that  specific  simulator.  After  using  all  three  systems,  subjects  completed  a 
second  post  questionnaire  (Survey  2),  which  asked  them  to  compare  all  three  systems  to  each  other.  The 
participant’s  performance  metrics  were  also  collected  from  each  of  the  simulators. 

RESULTS 

Demographics 

Subjects  were  categorized  as  Novice  (n=37),  Intermediate  (n=31),  or  Expert  (n=37).  Sixty-two  percent  of  subjects 
were  men  and  38%  were  women  with  an  average  age  of  43.  On  average,  participants  had  15  years  in  practice  and  3 
years  of  robotic  experience.  Seventy-six  percent  were  attending  physicians  and  73%  of  participants  were  currently 
or  had  received  robotic  training,  while  41%  provided  that  they  train  residents  and  fellows.  There  were  differences  in 
the  average  age  and  number  of  years  in  practice  of  participants  based  on  the  classification  of  expert,  intermediate  or 
novice  (number  of  robotic  procedures).  These  are  to  be  expected,  since  higher  ages  are  required  to  achieve  higher 
number  of  years  of  practice  and  larger  numbers  of  robotic  procedures. 

Validation 

The  types  of  validity  evaluated  in  this  experiment  were  face,  content,  and  construct.  To  analyze  the  systems  for  face 
validity  and  content  validity,  questions  from  Survey  1  were  used.  The  questions  were  evaluated  on  a  five  point 
Likert  scale  (Strongly  Disagree,  Disagree,  Neither  Agree  or  Disagree,  Agree,  and  Strongly  Agree).  Face  validity  was 
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analyzed  by  expert  and  intermediate  feedback  as  recommended  by  Van  Nortwick  et  al.  (2010)  because  these  are  the 
users  most  familiar  with  the  robotic  system;  however,  only  expert  feedback  was  used  for  content  validity  because 
they  have  the  best  ability  to  judge  the  appropriateness  of  the  system  as  a  training  tool.  For  construct  validity, 
performance  metrics  such  as  Overall  Score,  Time  to  Complete,  Number  of  Errors,  and  Economy  of  Motion  were 
analyzed  (Table  1). 


Table  1.  Questions  and  data  used  for  different  levels  of  validity. 


Type  of 
Validity 

Evaluation 

Type  of 
Participant 

Question/Metric 

Face 

Validity 

Survey  1 

Expert  and 
Intermediate 

Ql:  The  hand  controllers  on  this  simulator  are  effective  for  working 
in  the  simulated  environment  (Likert). 

Q4:  The  device  is  a  sufficiently  accurate  representation  of  the  real 
robotic  system  (Likert). 

Content 

Validity 

Survey  1 

Expert 

Q2:  The  3D  graphical  exercises  in  the  simulator  are  effective  for 
teaching  robotic  skills  (Likert). 

Q5:  The  scoring  system  effectively  communicates  my  performance 
on  the  exercise  (Likert). 

Q6:  The  scoring  system  effectively  guides  me  to  improve 
performance  on  the  simulator  (Likert). 

Construct 

Validity 

Simulator 

Experts  and 
Novices 

Overall  Score  (points) 

Number  of  Errors  (count) 

Time  to  Complete  (seconds) 

Economy  of  Motion  (centimeters) 

Face  Validity 

The  responses  of  Intermediate  and  Expert  participants  (n=68)  were  used  to  determine  face  validity  (Table  2).  A  Chi- 
square  test  of  independence  was  used  to  evaluate  the  distribution  of  scores  for  a  specific  simulator  in  relation  to  the 
order  of  the  system’s  presentation  to  the  subject.  This  analysis  indicated  that  there  was  no  difference  in  participants’ 
answers  according  to  the  order  in  which  the  systems  were  presented;  and  established  that  no  bias  was  present  due  to 
the  presentation  order  (p>0.05).  These  questions  asked  participants  to  evaluate  whether  the  hand  controllers  on  the 
simulator  were  effective  for  working  in  the  simulated  environment  (Question  1)  and  if  the  device  is  a  sufficiently 
accurate  representation  of  the  real  robotic  system  (Question  4).  For  both  questions,  the  RoSS  had  the  lowest  average 
score,  dV-Trainer  had  the  second  highest  score,  and  the  dVSS  had  the  highest  score  of  the  three.  A  repeated 
measures  ANOVA  verified  that  the  systems  were  scored  differently  for  both  questions  (p<0.001). 


Table  2.  Average  scores  from  a  5-point  Likert  scale  on  face  validity. 


DVSS 

dV-Trainer 

RoSS 

Ql:  The  hand  controllers  on  this  simulator  are  effective  for 
working  in  the  simulated  environment. 

4.80 

3.62 

2.17 

Q4:  The  device  is  a  sufficiently  accurate  representation  of 
the  real  robotic  system. 

4.65 

3.45 

1.82 

Content  Validity 

Expert  (n=34)  responses  were  used  to  determine  whether  the  simulators  were  appropriate  teaching  modalities  (Table 
3).  As  seen  in  Table  3,  100%  of  participants  either  agreed  or  strongly  agreed  that  the  3D  graphical  exercises  in  the 
dVSS  were  effective  for  teaching  robotic  skills  while  59%  disagreed  or  strongly  disagreed  that  the  RoSS’ 
capabilities  were  effective.  When  asked  if  the  scoring  system  effectively  communicated  their  performance,  88%  of 
dVSS  users  agreed  or  strongly  agreed,  while  79%  of  dV-Trainer  users  agreed  or  strongly  agreed.  Similarly,  91%  and 
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82%  of  participants  agreed  or  strongly  agreed  that  the  dVSS  and  dV-Trainer,  respectively,  effectively  guided  them 
to  improve  their  performance,  while  only  36%  felt  the  RoSS  provided  the  same  guidance. 


Table  3.  Scores  on  a  5  point  Likert  scale  for  content  validity  questions. 


Likert  Score 

Strong  Dis 

Disagree 

Neither 

Agree 

Strong  Agree 

Q2:  The  3D  graphical  exercises  in  the  simulator  are  effective  for  teaching  robotic  skills. 

DVSS 

0% 

0% 

0% 

35.3% 

64.7% 

dV-Trainer 

2.9% 

5.9% 

11.8% 

50.0% 

29.4% 

RoSS 

20.6% 

38.2% 

17.6% 

17.6% 

5.9% 

Q5:  The  scoring  system  effectively  communicates  my  performance  on  the  exercise. 

DVSS 

2.9% 

5.9% 

2.9% 

38.2% 

50.0% 

dV-Trainer 

2.9% 

2.9% 

14.7% 

55.9% 

23.5% 

RoSS 

17.6% 

20.6% 

26.5% 

29.4% 

5.9% 

Q6:  The  scoring  system  effectively  guides  me  to  improve  performance  on  the  simulator. 

DVSS 

0% 

0% 

8.8% 

61.8% 

29.4% 

dV-Trainer 

2.9% 

2.9% 

11.8% 

61.8% 

20.6% 

RoSS 

18.2% 

18.2% 

27.3% 

33.3% 

3.0% 

Construct  Validity 

The  overall  score,  number  of  errors,  time  to  complete,  and  economy  of  motion  scores  collected  by  the  simulators  for 
Experts  (n=37)  and  Novices  (n=37)  were  used  to  compare  construct  validity  (Table  4).  Overall  score  is  a  metric 
synthesized  by  multiple  metrics  and  is  specific  to  the  individual  simulator.  Intermediate  subjects  were  not  included 
in  the  construct  validity  analysis  because  it  was  only  necessary  to  look  if  the  simulator  could  distinguish  specifically 
between  novice  and  expert  users. 

For  the  RoSS,  the  analysis  has  23  missing  data  points  because  the  system  does  not  report  scores  when  a  user  exceeds 
a  maximum  exercise  time  or  chooses  to  terminate  the  exercise  before  completion.  This  resulted  in  a  sample  of  30 
experts  and  21  novices  on  that  system.  A  Mann- Whitney  U  test  showed  that  the  distributions  of  time  (p=0.221), 
number  of  errors  (p=0.644),  and  economy  of  motion  (p=0.566)  were  not  statistically  different  for  the  experts 
compared  to  the  novice  group.  The  overall  score  metric  is  not  automatically  exported  by  the  simulator  and  therefore 
was  not  analyzed  for  this  sytem. 

The  dV-Trainer  analysis  of  experts  (n=37)  and  novices  (n=37)  had  three  missing  values  for  economy  of  motion  and 
completion  time  and  five  for  the  overall  score  metric,  thus  the  analysis  contained  varying  number  of  subjects.  A 
Mann- Whitney  U  test  showed  that  the  distribution  of  the  overall  scores  was  not  significantly  different  for  the  expert 
compared  to  the  novice  group  (p=0.061).  These  tests  did  confirm  statistical  differences  for  economy  of  motion 
(p<0.001)  and  time  to  complete  (p<0.001)  for  this  system  with  a  lower  economy  of  motion  value  and  shorter 
completion  time  for  expert  users  compared  to  novices. 

The  dVSS  analysis  included  all  novice  (n=37)  and  expert  (n=37)  participants.  Using  a  Mann- Whitney  U  test,  time  to 
complete  (p<0.001)  and  overall  score  (p=0.006)  were  significantly  different  for  the  expert  compared  to  the  novice 
group.  The  expert  group  had  a  higher  score  and  a  shorter  completion  time  compared  to  the  novice  group.  However, 
economy  of  motion  did  not  show  a  statistical  difference  with  this  analysis  (p=0.216). 
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Table  4.  Mann- Whitney  U  test  level  of  significance  on  construct  validity  measures 


DVSS 

dV-Trainer 

RoSS 

Time  to  Complete 

p<0.001 

p<0.001 

p=0.221 

Overall  Score 

p<0.01 

p=0.061 

n/a 

Economy  of  Motion 

p=0.216 

p<0.001 

p=0.566 

Number  of  Errors 

n/a 

n/a 

p=0.644 

The  construct  validity  of  the  simulators  was  more  specifically  analyzed  in  terms  of  the  self-reported  number  of  cases 
of  all  participants  (n=105)  using  a  non-parametric  correlation  coefficient  (Spearman’s).  For  the  RoSS,  30 
participants  were  excluded  from  the  analysis.  For  the  participants  that  were  included  in  the  analysis  (n=75),  there 
was  not  a  significant  correlation  between  time  to  complete  (p=0.181),  number  of  errors  (p=0.563),  or  economy  of 
motion  (p=0.390)  with  the  total  number  of  robotic  cases  performed. 

For  the  dV-Trainer,  four  participants  were  excluded  from  the  entire  analysis  and  two  participants  were  excluded 
from  the  overall  score  (Overall  Score  n=99;  Economy  of  Motion  and  Time  to  Complete  n=101).  When  analyzing  the 
number  of  participants’  robotic  cases,  there  was  a  statistically  significant  correlation  between  overall  score  (p=0.03), 
economy  of  motion  (p<0.01),  and  time  to  complete  (p<0.01).  The  correlation  value  was  negative  for  economy  of 
motion  and  time  to  complete,  showing  that  with  a  greater  number  of  robotic  cases,  the  time  taken  and  distance 
moved  decreased.  The  correlation  was  positive  for  overall  score  indicating  that  the  participants’  score  increased  with 
the  number  of  robotic  cases  performed. 

For  the  dVSS,  two  participants  were  excluded  from  the  analysis  (n=103).  When  analyzing  the  metrics  in  terms  of  the 
total  number  of  robotic  cases  performed,  there  was  a  statistically  significant  difference  between  overall  score  (p 
=0.01)  and  time  to  complete  (p  <0.01).  The  correlation  value  was  negative  for  time  and  positive  for  overall  score, 
signifying  that  with  more  robotic  cases  the  time  taken  decreased  and  the  score  increased.  There  was  not  a 
statistically  significant  correlation  between  economy  of  motion  and  the  total  number  of  robotic  cases  performed 
(p=0.105). 


Table  5.  Correlation  between  level  of  experience  and  simulator  scores 


DVSS 

dV-Trainer 

RoSS 

Overall  Score 

p=0.001 

p=0.031 

n/a 

Time  to  Complete 

p<0.001 

p<0.001 

p=0.181 

Economy  of  Motion 

p=0.105 

p<0.001 

p=0.390 

Number  of  Errors 

n/a 

n/a 

p=0.563 

Usability  (Preference) 

The  questions  from  the  Survey  2  were  used  to  understand  the  preference  of  the  subjects  when  using  the  simulators. 
All  subjects  were  included  in  this  analysis  except  for  two  participants  who  were  dropped  from  the  analysis  because 
they  did  not  complete  the  questionnaire.  The  participant’s  responses  to  the  usability  questions  can  be  seen  in  Figure 
3: 


•  If  you  are  (were)  a  program  director,  which  simulator  would  you  choose  for  your  trainees; 

•  In  which  simulator  were  you  physically  more  comfortable; 

•  Which  simulator  had  the  best  hand  controls; 

•  Which  simulator  had  the  best  foot  controls; 

•  Which  simulator  had  the  best  3D  vision; 

•  Were  you  feeling  stressed  or  annoyed  by  any  of  the  simulators? 
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Usability 
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Director  Comfort  Controls  Controls  or  Stressed 


“dVSS 
a  dV-Trainer 
□RoSS 

_  ^dVT+RoSS 


Figure  3.  Description  of  usability  responses 


Overall,  most  participants  preferred  the  dVSS  and  indicated  that  they  would  choose  this  device  as  a  training  system 
if  they  were  a  program  director.  Participants  not  only  felt  most  comfortable  in  the  dVSS,  but  also  felt  that  the  system 
had  the  best  control  and  vision  equipment.  The  least  preferred  system  was  the  RoSS  which  most  participants  also 
agreed  made  them  feel  stressed  or  annoyed.  Ten  percent  of  participants  also  responded  that  they  felt  stressed  or 
annoyed  by  both  the  dV-Trainer  (dVT)  and  the  RoSS. 

Cost 


All  participants  were  also  asked  to  provide  feedback  on  their  simulator  preference  in  terms  of  the  cost  of  the  system. 
The  responses  were  analyzed  in  terms  of  the  frequency  of  the  responses  given.  Most  participants  felt  that  the  mimic 
dV-Trainer  was  worth  the  investment;  while  most  felt  that  the  RoSS  was  not  worth  the  money.  When  asked  about 
the  dVSS,  only  56%  of  participants  agreed  that  it  was  worth  the  investment.  Figure  4  provides  a  full  description  of 
the  responses. 


Worth  the  Investment? 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

**The  price  presented  in  the  survey 

dVSS  dV-Trainer  RoSS  included  all  necessary  support 

($600,000)  ($100,000)  ($100,000)  equipment  to  make  the  simulator 

Figure  4.  Description  of  cost  preferences 
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DISCUSSION 

The  aim  of  this  study  was  to  conduct  a  comparison  of  the  three  commercially  available  simulators  used  to  train 
surgeons  on  the  daVinci  robotic  system.  The  study  was  performed  for  the  US  Army  to  assist  them  in  making  a 
purchasing  and  deployment  decision  regarding  robotic  simulators.  Their  interest  is  in  re-training  robotic  surgeons 
who  have  been  deployed  to  combat  zones,  where  they  have  served  as  trauma  surgeons  for  many  months.  Prior  to 
resuming  their  robotic  specialties,  these  surgeons  need  a  program  to  both  refresh  and  re- validate  their  robotic  skills. 
This  study  provided  information  about  the  face,  content,  and  construct  validity  as  well  as  usability  of  the  systems. 
The  simulators  were  perceived  to  be  different  in  their  representation  of  the  real  robotic  system.  The  dVSS  was  most 
preferred  in  terms  of  ergonomics  and  usability;  however,  most  participants  did  not  feel  that  this  system  was  worth  a 
$600,000  investment.  In  terms  of  cost,  most  participants  agreed  that  the  dV-Trainer  had  the  best  cost-effectiveness. 
The  RoSS  was  the  least  preferred  system  for  comfort  and  other  usability  aspects  (i.e.,  hand  controls,  foot  controls, 
and  3D  interface),  with  most  participants  feeling  stressed  or  annoyed  when  using  the  system.  This  study  was  unable 
to  validate  the  face,  content,  or  construct  validity  for  this  system. 

The  dVSS  leverages  the  actual  hardware  used  to  perform  robotic  surgeries  for  use  in  the  simulated  environment, 
which  allows  for  a  more  realistic  experience,  but  decrease  its  availability  and  creates  a  higher  cost  for  training  than 
other  robotic  simulators.  Economy  of  motion  was  not  able  to  differentiate  novices  from  experts  in  the  dVSS,  which 
could  be  attributed  to  the  ease  of  use  of  the  controllers  allowing  novices  to  move  the  controls  as  efficiently  as 
experts.  The  generous  workspace  of  the  dVSS  could  also  have  an  impact  on  the  lack  of  difference.  In  contrast  to  the 
dVSS,  the  dV-Trainer  is  a  standalone  simulator  and  does  not  require  the  support  of  the  daVinci  hardware  to  operate. 
This  allows  for  better  accessibility  and  requires  less  of  an  investment  for  training.  The  overall  score  aspect  of 
construct  validity  may  not  have  shown  a  difference  between  novices  and  experts  because  of  the  way  that  the  scoring 
is  developed.  The  scoring  system  is  constructed  with  a  “ceiling”  that  prevents  users  from  achieving  a  high  overall 
score  without  attaining  high  scores  across  multiple  metrics. 

Currently,  there  is  limited  data  available  that  confirms  construct  validity  of  the  RoSS.  Similarly  to  Raza  (2013),  this 
study  was  unable  to  confirm  a  difference  between  experts  and  novices  in  terms  of  time  taken  to  complete  the 
exercise.  Time  to  complete,  as  well  as  economy  of  motion,  is  considered  a  highly  relevant  measurement  of  expertise 
levels  for  robotic  surgeons  (Perrenot,  Perez,  Tran,  Jehl,  Felb linger,  Bresler,  &  Hubert,  2012).  To  our  knowledge  this 
three-part  study  is  the  first  to  compare  all  three  available  systems.  This  study  involved  the  largest  sample  size  and 
diversity  of  participants  (i.e.,  experience  levels,  number  of  robotic  cases,  and  subspecialty  type)  thus  far  in  relevant 
publications.  The  lack  of  consistency  in  the  available  exercises  and  scoring  systems  across  the  three  systems  was  a 
limitation  to  the  study.  Considerations  for  future  research  would  be  to  use  more  complex  exercises  and  increase  the 
depth  of  the  face  and  content  validity  evaluation. 

Current  research  is  focused  on  the  effectiveness  of  the  simulators  and  objectively  measuring  the  transfer  of  training 
to  the  actual  robotic  system.  All  three  simulators  will  be  examined  in  this  final  stage  of  the  experiment;  however,  the 
results  of  this  three-part  study  will  guide  the  choice  of  simulators  used  for  future  studies  at  Florida  Hospital 
Nicholson  Center  and  may  also  influence  decisions  at  other  laboratories.  Also,  this  research  may  impact  the 
purchasing  decisions  of  customers  for  these  devices. 
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ABSTRACT 

The  daVinci  Surgical  System  offers  surgeons  improved  capabilities  for  performing  complex  minimally  invasive 
procedures;  however,  there  is  no  standardized  assessment  of  robotic  surgeons  and  a  need  exists  to  ensure  that  a 
minimal  standard  of  care  is  provided  to  all  patients.  The  Department  of  Defense  and  governing  surgical  societies 
convened  consensus  conferences  to  develop  a  national  initiative,  resulting  in  a  curriculum  called  the 
Fundamentals  of  Robotic  Surgery  (FRS).  FRS  is  comprised  of  an  online  curriculum  and  a  psychomotor  skills 
dome. 

This  paper  describes  the  production  process  used  to  create  a  psychomotor  skills  assessment  device  -  the  FRS 
Dome.  The  device  was  designed  to  measure  the  essential  skills  that  are  required  of  any  robotic  surgeon  and  to 
provide  a  basis  upon  which  to  grant  or  deny  privileging  with  the  robot.  It  was  constructed  to  test  seven  tasks  of 
manual  dexterity:  Docking,  Ring  Tower  Transfer,  Knot  Tying,  Suturing,  4th  Arm  Cutting,  Puzzle  Piece 
Dissection,  and  Energy  Dissection. 

The  initial  design  of  the  device  was  created  by  a  committee  of  experienced  minimally  invasive  surgeons,  with  a 
background  in  testing  protocols  and  materials.  The  design  was  rendered  in  computer  animation,  which  kick- 
started  a  prototyping  effort  with  physical  materials.  These  included  platinum  cure  silicone  approximating  human 
tissue  and  a  3D  polyjet  printer  for  the  structural  framework.  Usability  testing  was  conducted  and  iterative 
modifications  were  made  to  improve  ergonomics,  standardization,  and  cost  requirements.  Final  CAD  diagrams 
and  specifications  were  created  and  distributed  to  medical  and  simulation  companies  for  both  physical  and 
digital  manufacturing.  This  development  process  demonstrates  the  evolution  of  a  simulation  and  a  physical 
testing  device  based  on  international  expert  consensus.  The  specifications  are  open  source,  allowing  competitive 
production  and  future  iterations.  The  goal  of  this  paper  is  to  discuss  how  this  device  evolved  from  an  idea  to  a 
manufactured  product  and  a  digital  simulation. 
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INTRODUCTION  AND  BACKGROUND 

Robotic  surgery  has  been  established  as  an  innovative  approach  in  surgery  due  to  a  telemanipulator  device, 
which  introduced  a  new  dimension  into  surgical  tools.  This  device  allows  surgeons  to  manipulate  robotic  arms 
from  a  remote  console  to  perform  complex  surgical  procedures.  Robotic  surgical  systems  overcome  laparoscopic 
limitations  and  facilitate  the  performance  of  minimally  invasive  surgery  due  to  3D  vision,  7-degree-of-freedom 
instruments,  tremor  abolition,  motion  amplification,  and  stabilization  of  the  camera  (Patel  et  al.,  2013;  Hubens, 
Coveliers,  Balliu,  Ruppert,  &  Vaneerdeweg,  2003;  Blavier,  Gaudissart,  Cadiere,  &  Nyssen,  2007).  The  system 
also  offers  lOx  magnification,  wristed  instruments,  and  a  third  working  arm.  Currently,  the  only  system  is 
Intuitive’s  da  Vinci  Surgical  System  (Figure  1). 


Figure  1.  da  Vinci  Surgical  System 


Robotic  surgery  has  demonstrated  safety  and  effectiveness  for  urologic,  gynecologic,  ENT,  and  complex  general 
surgery  procedures  (Barbash,  Friedman,  Glied,  &  Steiner,  2014;  Serati  et  al.,  2014;  Maan,  Gibbins,  Al-Jabri,  & 
D’Souza,  2012;  Luca  et  al.,  2013;  Zureikat  et  al.,  2013).  Exponential  growth  of  minimally  invasive  procedures, 
particularly  robotic-assisted  procedures,  raises  the  question  of  how  to  assess  robotic  surgical  skills.  This  device 
also  introduces  a  specific  need  for  training  and  certification  to  ensure  a  minimal  standard  of  care  for  all  patients. 
Some  institutions  have  attempted  to  develop  and  validate  robotic  training  in  regards  to  specific  specialties 
(Chitwood  et  al.,  2001;  Geller,  Schuler,  &  Boggess,  2011;  Grover,  Tan,  Srivastava,  Leung,  &  Tewari,  2010; 
Chowriappa  et  al.,  2014;  Jarc  &  Curet,  2014);  however,  the  lack  of  a  national  standard  has  pushed  surgical 
societies  (e.g.  the  Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons  and  Society  of  Robotic 
Surgery)  to  develop  a  unified  approach  and  standard  for  robotic  skills  training  (Zorn  et  al.,  2009). 


To  develop  a  comprehensive  model  for  robotic  surgery,  the  Department  of  Defense,  Veterans  Administration, 
and  fourteen  surgical  specialty  societies  convened  multiple  consensus  conferences  to  create  the  Fundamentals  of 
Robotic  Surgery  (FRS)  curriculum.  A  similar  education  and  training  initiative  was  implemented  for  use  in 
laparoscopic  surgery,  which  resulted  in  the  Fundamentals  of  Laparoscopic  Surgery  (FLS).  FRS  Conference 
participants  included  more  than  80  subject  matter  experts  (SMEs),  consisting  of  surgeons,  psychologists, 
engineers,  simulation  experts,  and  medical  educators  (Smith,  Patel,  Chauhan,  &  Satava,  2013). 

The  committee’s  vision  of  FRS  was  driven  by  two  main  goals:  to  ensure  a  perfect  understanding  of  the  basics  of 
robotic  surgery  and  to  develop  a  psychomotor  skills  program  that  focused  on  basic  robotic  tasks.  The  intended 
users  for  this  program  are  novice  robotic  surgeons,  who  could  be  residents  or  fellows  and  attending  surgeons 
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who  have  never  used  the  robotic  system.  The  committee  began  by  outlining  outcomes  measures  and  metrics, 
which  touched  on  the  essential  cognitive,  psychomotor,  and  team  training  skills.  This  resulted  in  a  prioritized 
matrix  of  25  robotic  surgery  concepts,  which  is  the  core  material  used  in  the  design  and  development  of  the  FRS 
Curriculum  (Smith,  Patel,  Satava  R,  2013).  Two  assessment  tools  were  created:  an  online  curriculum  for 
knowledge  and  team  training  skills  and  a  device  for  psychomotor  skill  training  and  evaluation  (  Levy,  n.d.). 

This  paper  discusses  the  process  for  designing  and  creating  the  physical  device,  known  as  the  FRS  dome.  The 
purpose  is  to  share  the  evolution  of  an  idea  to  a  usable  device.  The  dome  was  conceived  by  experts  who 
identified  a  clear  need  for  robotic  education  and  collectively  developed  a  solution  to  fill  the  gap.  The  medical 
field  is  a  constant  progression  of  new  concepts,  devices,  and  technology.  This  paper  also  outlines  the  framework 
for  which  others  can  develop  and  introduce  new  concepts  in  medicine  and  other  domains. 


BRAINSTORMING  AND  CONCEPT  DEVELOPMENT 
Exercise  Development 

Of  the  25  FRS  concepts,  16  are  directly  linked  with  psychomotor  skills.  The  FRS  committee  members  then 
identified  seven  exercises  that  incorporated  all  16  skills.  These  exercices  include  docking  and  instrument 
insertion,  tower  transfer,  knot  tying,  railroad  track,  4th  arm  cutting,  puzzle  piece  dissection,  and  vessel  energy 
dissection  (Table  1).  Docking  and  instrument  insertion  is  an  essential  and  unique  robotic  skill  to  begin  a 
procedure.  Failure  at  this  stage  of  the  procedure  can  compromise  the  surgery.  Ring  Tower  transfer  is  a  non- 
surgical  exercise  that  introduces  the  utilization  of  endowrist  manipulation  and  the  7  degrees  of  freedom  to 
surgeons.  Knot  tying  and  railroad  track  are  the  base  of  a  suturing  exercise.  The  technology  introduced  in  the 
wristed  instruments  facilitates  the  performance  of  these  tasks.  4th  arm  cutting  is  another  task  specific  to  robotics, 
which  tests  surgeon’s  autonomy.  The  4th  arm  allows  surgeons  to  manage  three  instruments  by  using  a  foot  pedal 
to  switch  between  working  arms.  Puzzle  piece  and  vessel  energy  dissection  are  critical  tasks  which  incorporate 
complex  articulation  of  instruments  and  application  of  energy  (i.e.  cauterization  and  cutting). 


Table  1:  Description  of  the  basic  psychomotor  skills  attached  to  the  seven  FRS  tasks. 


Exercises 

Skills 

Taskl:  Dockin 

g  &  Instrument  In 

.  .  ^ 

sertion: 

-  Docking 

-  Instrument  insertion 

-  Eye-hand  coordination 

-  Operative  field  of  view 

Task  2:  Ring  Tower  Transfer: 

I'M* 

-  Eye-hand  coordination 

-  Camera  navigation 

-  Clutching 

-  Wrist  articulation 

-  A-traumatic  handling 

Task  3:  Knot  T 

ying: 

U-ir. 

-  Knot  tying 

-  Suture  handling 

-  Eye-hand  coordination 

-  Wrist  articulation 

Task  4:  Railroe 

id  Track: 

Wm 

-  Needle  handling  &  manipulation 

-  Wrist  articulation 

-  A-traumatic  handling 

-  Eye-hand  coordination 

Task  5:  4th  Arm  Cutting: 

-  Multiple  arm  control  &  switch 

-  Cutting 

-  A-traumatic  handling 

-  Eye-hand  coordination 
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Task  6:  Puzzle  Piece  Dissection: 

-  Sharp  and  blunt  dissection 

\\  " 

-  Cutting 

j  J 

-  A-traumatic  handling 

-  Eye-hand  coordination 

V 

-  Wrist  articulation 

Task  7:  Vessel 

Energy  Dissection: 

-  Energy  sources  use 

-  Sharp  dissection 

fr  M  \| 

-  Cutting 

4  ■  1 

-  Multiple  arm  control 

V  J 

-  A-traumatic  handling 

-  Eye-hand  coordination 

Device  Development 

The  FRS  committee  envisioned  all  of  the  exercises  contained  on  the  outer  surface  of  a  single  device.  This  would 
allow  for  the  exercises  to  be  administered  quickly  and  easily,  incur  less  cost,  and  ensure  uncomplicated  storage 
and  transportation.  The  semi- spherical  form  (i.e.  the  dome),  was  quickly  decided  on  as  a  shape  which  would 
integrate  with  the  current  robotic  system.  They  depicted  their  ideas  through  simple  drawings  and  crude  models 
made  from  materials  found  on  hand.  During  initial  design  planning,  conference  participants  experimented  with  a 
variety  of  arrangements  of  the  exercises  on  the  dome. 

A  final  sketch  was  developed  and  delivered  to  a  3D  digital  artist  to  create  static  pictures  of  the  device,  along  with 
an  animation  of  the  performance  of  each  exercise.  The  CGI  provided  the  first  formal  images  of  the  dome,  which 
gave  life  to  the  device  and  proved  feasibility.  The  realistic  animations  showed  the  exercises  being  performed  and 
gave  committee  members  a  visual  concept  of  how  the  device  would  function  (Figure  2). 


Figure  2.  The  initial  3D  graphic  FRS  dome  design 


PROTOTYPING 

The  prototyping  process  began  using  the  ideas  developed  in  the  design  meeting  and  the  CGI.  This  process  would 
prove  to  be  fundamental  in  confirming  the  design  expectations.  It  was  essential  to  determine  if  a  single  device 
could  physically  house  all  of  the  exercises  effectively,  if  the  planned  architecture  was  compatible  with  the 
robotic  system,  and  if  the  outcomes  of  the  exercises  could  be  measurable  and  reproducible. 

Low-fidelity  Prototypes 

Low-fidelity  prototypes  (LFPs)  were  created  using  simple  and  inexpensive  materials.  None  of  the  materials  used 
in  the  LFPs  were  intended  for  inclusion  in  a  final  product.  These  materials  were  chosen  because  they  were 
readily  available,  inexpensive,  and  easy  to  manipulate  to  test  fit  and  function.  These  materials  allowed  rapid  trial 
and  error  testing  of  the  technical  aspects,  clarifying  requirements,  and  proving  usability.  The  testing  of  the  LFPs 
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was  performed  using  the  da  Vinci  Surgical  System  and  was  video  recorded.  These  recordings  were  sent  to  FRS 
committee  members  to  provide  their  feedback.  Each  LFP  resulted  in  multiple  improvements  to  the  designs, 
which  were  tested  on  subsequent  prototype  versions. 

The  base  model  of  the  LFPs  was  created  using  half  of  an  8”  Styrofoam  sphere  as  the  support  structure,  yellow 
felt  material  as  the  fat  layer,  a  latex  swimming  cap  for  the  skin  layer,  and  straws  for  the  embedded  vessels.  The 
base  of  the  towers  was  constructed  using  synthetic  foam  blocks  carved  into  a  cone  shape  (Figure  3).  The  exercise 
patterns  were  drawn  onto  the  surface  using  a  permanent  marker. 


Figure  3.  Base  of  Low  Fidelity  Prototypes 

The  LFPs  evolved  over  six  iterations,  all  of  which  introduced  design  improvements  (Figure  4).  At  the  earliest 
phase  in  LFP  testing,  it  was  quickly  realized  that  the  dome  size  was  too  large  to  fit  under  the  robot  arms 
appropriately.  So,  the  dome  size  was  decreased  from  8”  to  7”.  Another  modification  made  early  in  the  LFP 
development  was  to  change  the  4th  arm  cutting  band  from  a  rigid  tube  to  an  elastic  band.  This  allowed  for  the 
user  to  adequately  stretch  the  band  prior  to  each  cut. 


Figure  4.  Iterations  of  LFPs 


The  suturing  and  dissection  exercises  involved  the  most  modifications  during  the  LFP  stages.  The  original 
cloverleaf  shaped,  used  for  the  dissection  exercise,  was  found  to  be  too  large  and  did  not  allow  for  the  surgeons 
to  access  the  section  of  the  shape  that  was  located  on  the  backside  of  the  dome.  The  size  of  the  pattern  was 
reduced;  however,  this  did  not  mitigate  the  accessibility  issue.  The  team  experimented  with  other  options,  such 
as  splitting  the  clover  leaf  into  three  sections  and  adding  smaller  shapes  to  the  center  of  the  cutting  area.  This 
design  was  not  practical  because  once  the  smaller  shapes  were  cut,  the  latex  receded  and  inhibited  surgeons  from 
cutting  the  surrounding  shape. 

Eventually,  the  dissection  shape  evolved  to  a  puzzle  piece  which  incorporated  all  of  the  prerequisites  for  the 
dissection  exercise  (i.e.  an  accessible  shape  and  a  complex  design).  By  using  this  compact  pattern  it  became 
clear  that  all  exercises  could  be  grouped  into  an  area  covering  only  one  third  of  the  surface  of  the  dome.  This 
opened  the  opportunity  to  replicate  the  cluster  of  exercises  three  times  on  the  surface,  reducing  the  materials  and 
costs  for  repeatedly  practicing  with  the  device.  Another  obstacle  was  to  build  the  suturing  exercise  with  the 
adequate  materials  and  placements,  to  ensure  a  realistic  feeling  of  suturing.  Originally,  the  incision  was  made 
into  the  latex  swim  cap,  however  the  latex  would  tear  away  and  recede  after  the  incision  was  cut  in  this  model. 
Two  versions  of  the  suture  module  were  experimented  with:  an  embedded  silicone  and  an  external  latex  model. 
Eventually  an  embedded  silicone  model  was  chosen  as  the  most  realistic  and  practical  for  the  exercise. 
Ultimately  the  basic  structural  changes  found  in  the  low- fidelity  prototyping  were: 

•  The  dome  base  needed  to  be  reduced  to  7” 

•  The  dome  base  needed  to  be  substantial  in  weight  to  keep  from  moving  under  the  force  of  the  robot 

•  A  smaller,  yet  equally  complex  dissection  shape  was  necessary 

•  The  exercises  set  could  be  grouped  to  allow  it  to  be  repeated  on  the  surface  of  a  single  dome 
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•  The  magnets  which  held  the  towers  to  the  dome  needed  to  be  of  sufficient  strength  to  hold  through  the 
layers  of  fat  and  skin 

High-Fidelity  Prototypes 

The  high-fidelity  prototypes  (HFPs)  were  made  using  higher  quality,  custom  materials.  These  materials  had  the 
desired  qualities  of  the  final  product  and  could  be  used  as  a  basis  for  the  large  scale  manufacturing  process.  The 
styrofoam  base  from  the  LFPs  was  replaced  with  a  support  structure  that  was  printed  using  a  3D  polyjet  printer 
(Figure  5).  A  polyjet  type  3D  printer  works  similarly  to  an  inkjet  printer  in  that  it  distributes  layers  of  polymer  to 
build  the  desired  design,  which  is  cured  by  UV  light.  This  type  of  printer  was  chosen  because  of  the  versatility 
allowed  by  printing  multiple  materials  at  once.  Also,  the  jet  lays  16/ira  layers  of  liquid  polymer  which  gives 
printed  parts  a  finer  resolution.  Using  this  printer,  a  dome  shell  with  a  lid  was  created.  The  shell  and  lid  had 
divots  covering  the  surface  which  allowed  for  magnets  to  be  moved  to  many  different  placements  on  the  dome 
during  design  experiments.  A  small  jig  was  also  created  using  the  3D  printer.  Prior  to  the  creation  of  the  jig,  the 
wires  were  made  by  hand,  but  the  jig  enables  the  standardized  creation  of  the  S-shaped  and  I-shaped  tower  wires. 
The  price  to  print  these  items  was  approximately  $1,000. 


Figure  5.  3D  printer  with  3D  printed  dome,  cap,  towers,  and  jig 


The  synthetic  tissue  layers  were  created  using  Smooth-On  platinum  cure  silicone  products.  These  are  two  part 
silicones  that  can  be  colored  and  mixed  with  other  additives  to  achieve  the  desired  product  attributes,  such  as 
durometer.  The  silicone  used  for  the  “fat”  layer  gave  a  gel-like  and  slightly  sticky  texture  (Eco-flex  Gel),  while 
the  “skin”  silicone  had  a  more  firm  and  non-sticky  quality  (Ecoflex-0030).  These  silicones  were  chosen  because 
they  gave  the  closest  resemblance  to  actual  tissue  properties.  The  fat  silicone  was  poured  directly  onto  the  dome 
to  the  desired  thickness.  A  clay  mold  was  then  made  to  replicate  that  thickness,  which  was  used  to  form  the  skin 
layer  (Figure  6).  Embedded  in  the  skin  was  a  layer  of  polyester  mesh,  which  helped  to  provide  structure  and 
stability  of  the  skin.  Small  vessels  were  also  created  by  quickly  curing  the  silicone  to  a  small  tube.  Using  these 
materials  we  were  able  to  create  a  set  of  synthetic  tissues  for  less  than  $20. 


Figure  6.  Pouring  of  silicones  and  first  HFP 


The  puzzle  piece  shape  and  the  other  markers  were  drawn  on  the  skin  surface  using  a  permanent  marker.  The 
exercises  were  drawn  on  in  different  locations,  sizes,  and  orientations  for  the  first  HFP.  After  testing  the  HFP  on 
the  robotic  system,  we  finalized  the  size  and  orientation  of  the  exercises  on  this  new  dome.  This  is  important 
because  as  learned  in  the  LFP  stage,  the  exercises  needed  to  be  placed  strategically  to  compensate  for  the  range 
of  movement  of  the  robotic  arms.  Despite  having  7  degree-of- freedom  instruments,  there  are  still  limitations  to 
the  amplitude  of  the  movement  of  the  robotic  arms.  We  also  determined  that  three  trials  of  each  exercise  could 
fit  on  one  dome,  so  each  work  station  (i.e.  group  of  exercises)  repeated  at  120  degree  increments  on  the  dome. 
Eventually,  we  determined  that  after  dissecting  the  three  vessels  significant  space  was  available  for  more 
dissection  in  the  fat  layer.  So,  we  added  three  additional  vessels  located  to  the  right  of  the  original  vessels  and 
out  of  range  of  potential  damage  from  other  exercises  (Figure  7).  By  doing  so,  the  fat  could  be  used  six  times 
and  the  skin  used  three  times,  which  incurs  less  of  a  manufacturing  cost  and  ultimately  training  costs. 
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Figure  7.  Vessel  placement  on  dome  and  in  fat 


Over  many  iterative  models,  we  improved  our  techniques  and  experimented  with  different  materials  and 
additives  to  achieve  the  desired  qualities.  For  example  we  began  adding  a  Thixotropic  additive  to  thicken  the 
mixture  and  allow  us  to  cast  the  material  onto  a  curved  surface.  We  also  tested  different  inks  and  techniques  of 
printing  the  shapes  and  markers  on  the  skin.  However,  most  inks  and  paints  could  not  be  used  on  silicone,  so  we 
decided  to  use  a  silicone  based  paint  product  which  cured  the  design  to  the  silicone  surface. 

We  3D  printed  miniature  dome  models  (2”  in  diameter)  to  begin  testing  molding  materials.  We  created  silicone 
molds  and  used  a  urethane  plastic  to  cast  the  model.  By  doing  this  we  realized  that  the  original  3D  printed 
material  was  porous  and  caused  bubbling  in  the  molding,  leading  to  surface  bubbles  on  casted  models.  So,  a  new 
full  sized  dome  was  printed  in  a  smoother  and  less  porous  material  which  would  be  better  for  manufacturing. 
The  new  dome  shell  and  cap  was  designed  with  divots  only  at  the  locations  necessary  for  holding  a  tower 
(Figure  8). 


Figure  8.  Final  3D  printed  dome  shell 


Since  this  device  will  be  used  for  training  and  education,  a  high  level  of  standardization  is  necessary.  For  this  we 
added  small  markers  which  ensure  that  the  pieces  are  assembled  correctly  and  in  a  standardized  manner  for  all 
participants.  Table  3  details  the  standardization  pieces. 


Table  3.  Description  of  the  Standardization  Markers 


Standardization  Markers 

T 

a  jl 

j- 1 

o 

IS 

Used  to  orient  the  towers  in  the  correct  direction  for  each  exercise. 

1 

Mangle  in  li< 

r  5 

L  \  / 

i 

Used  to  show  proper  orientation  of  the  towers  that  are  placed  in  the  cap. 
The  towers  are  placed  in  the  two  locations  directly  in  line  with  the 
puzzle  piece  and  with  the  tower  tongues  on  the  corresponding  line  of 
the  triangle.  This  ensures  that  the  S-shaped  towers  face  the  correct 
direction  for  all  users. 

Tower  orientation  markers 

These  markers  are  used  to  show  the  placement  of  the  towers  on  the  skin 
and  the  orientation  of  the  tower.  The  towers  are  placed  on  the  marker 
with  the  tongue  aligned  with  the  tongue  mark.  This  ensures  that  all 
towers  face  the  correct  way. 

* 

j 

Triangles  on  dome  shell 

These  small  markers  are  located  at  120  degree  increments  on  the  lower 
edge  of  the  dome.  They  signify  where  the  embedded  vessels  should  be 
located  when  the  tissue  layers  are  placed  on  the  shell. 
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1 

rriangles  on  fa 

t 

1 

There  are  two  types  of  triangle  markers  on  the  fat:  open  and  closed.  The 
closed  triangles  indicate  the  location  of  the  first  use  vessels.  When  the 
fat  is  placed  on  the  dome,  the  closed  triangle  is  aligned  with  the  triangle 
marker  on  the  dome  shell.  After  all  three  vessels  are  used,  the  fat  is 
rotated  and  the  open  triangles  are  aligned  with  the  triangles  on  the 
dome.  This  ensures  that  the  vessels  are  in  the  accurate  location  for  the 
dissection  exercises. 

T 

riangles  on  ski 

A 

n 

The  triangle  markers  on  the  skin  are  aligned  with  the  triangles  on  the  fat 
layer.  These  ensure  that  the  puzzle  piece  lies  directly  over  the  vessel  and 
that  the  tower  markers  align  with  the  underlying  magnets. 

Caj 

p  placement  no 

JS 

tch 

The  notch  in  the  cap  ensures  that  users  place  the  cap  in  the  correct 
orientation.  Since  the  magnet  divots  are  placed  in  the  shape  of  a 
triangle,  the  cap  has  to  be  secured  in  a  specific  orientation  for  the 
magnet  divots  to  align  properly. 

In  the  final  HFP,  the  exercises  existed  as  they  would  in  the  manufacturing  phase.  Final  testing  was  performed  in 
order  to  ensure  that  all  specifications  were  correct  and  to  build  a  specifications  document,  which  was  used  to 
create  final  CGI  and  CAD  files  (Figure  9). 


r  *  ) 


Figure  9.  Final  HFP 


PRODUCTION 

The  final  CGI,  CAD,  and  specification  document  were  sent  to  the  manufacturing  company  and  simulation 
companies  to  assist  them  in  their  development  of  physical  and  virtual  domes  (Figure  10). 
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Figure  10.  Final  CGI 

A  local  manufacturer,  familiar  with  the  materials  used  during  prototype  testing,  used  the  dome  and  performed  all 
of  the  exercises  prior  to  beginning  the  process.  This  provided  a  first-hand  experience  of  why  certain  material 
qualities  were  so  important.  The  goals  for  this  phase,  in  addition  to  mass  production,  were  to  maintain  device 
integrity  and  minimize  cost.  Some  of  the  materials  used  during  prototyping  were  more  expensive  than  what 
would  be  feasible  for  training  centers.  For  example  the  $1,000  materials  cost  for  the  3D  printed  dome  was 
reduced  to  less  than  $25. 

The  simulation  exercises  of  the  FRS  dome  will  be  incorporated  into  two  simulators:  the  da  Vinci  Skills 
Simulator  (dVSS)  and  the  Mimic  dV-trainer.  The  dVSS  is  a  simulation  system  which  integrates  with  the  actual 
console  of  the  surgical  system.  This  allows  users  to  train  using  the  exact  hardware  that  they  use  when  operating. 
The  dV-trainer  is  a  standalone  system  that  uses  custom  hardware  and  software.  To  ensure  that  the  simulated 
dome  acted  similarly  to  the  actual  dome,  the  research  team  evaluated  the  exercises  in  terms  of  qualities  such  as 
elasticity  of  materials  and  effects  of  excess  force  on  the  device.  Maintaining  the  physical  properties  of  the  dome 
was  paramount.  Since  the  simulations  will  be  unproctored,  these  features  are  important  in  designing  the  metrics. 


CONCLUSION 

Over  the  course  of  two  years,  we  created  an  easily  integrated  device,  using  low  cost  but  high-quality  materials. 
This  paper  outlines  the  steps  of  the  FRS  dome  from  idea  conception  to  the  development  of  physical  and  virtual 
devices.  The  goal  of  this  paper  is  to  share  the  evolution  and  process  for  others  interested  in  training  and 
assessment  devices.  Since  the  FRS  dome  specifications  are  open-source,  this  also  serves  as  an  important 
resource  for  potential  producers. 

We  have  taken  away  several  lessons  from  our  experimentation  that  made  our  process  a  success  including  having 
a  multidisciplinary  team,  soliciting  frequent  feedback,  using  easily  adaptable  designs,  testing  on  small  models, 
and  using  commercial  materials  during  prototyping.  Our  multidisciplinary  team  of  surgeons  and  engineers 
allowed  for  a  diverse  perspective  during  the  construction  of  the  device.  The  design  changed  many  times  and  it 
was  beneficial  to  start  off  using  basic  models  that  accommodated  the  varying  designs.  It  was  advantageous  to 
work  with  actual  manufacturing  materials  once  we  developed  a  functional  prototype  to  better  envision  the  final 
product  and  allow  a  smoother  transition  to  the  manufacturing  phase.  We  recommend  testing  materials  on  small 
models,  which  will  help  cut  time  and  costs.  Finally  if  possible,  work  closely  with  the  manufacturing  teams  at  an 
early  stage  of  development,  particularly  when  working  with  virtual  models.  This  will  help  to  flesh  out  details  and 
encourage  collaborative  development  earlier  in  the  process. 

The  next  step  of  this  work  is  to  conduct  formal  validation  testing  of  the  curriculum  including  the  device  and 
related  simulations  via  a  pilot  and  national  multi-site  validation  study.  The  FRS  dome  features  basic  robotic 
surgical  skill  exercises,  which  are  applicable  to  most  specialties.  This  basic  device  is  scalable  and  will  be  the 
foundation  for  the  future,  more  specialized  FRxS  devices  (e.g.,  the  Fundamentals  of  Robotic  Gynecologic 
Surgery  (FRGS)  and  the  Fundamentals  of  Robotic  Urologic  Surgery  (FRUS)). 
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Learning  Objectives: 

At  the  completion  of  this  presentation,  participants  should  be  better  able  to: 

1)  Understand  the  effect  of  latency  on  performance  in  robotic  simulation 

2)  Understand  the  current  technological  limitations  to  telesurgery 

Introduction  and  Objective: 

Robot-assisted  surgical  technology  has  been  used  for  remote  telesurgery  under  unique 
controlled  conditions.  However,  an  objective  scientific  measurement  of  the  effects  of 
latency  on  performance  and  the  feasibility  of  performing  procedures  with  standard 
existing  networks  has  not  been  previously  conducted.  This  research  measured  the 
effects  of  various  levels  of  latency  on  successful  performance  of  procedures  in  a 
simulator,  and  further  measured  the  real  latency  for  data  transfer  within  a  multi-campus 
metropolitan  hospital  system.  Together,  these  identify  whether  telesurgery  can  be  safely 
performed  in  an  existing  multi-campus  hospital  environment. 

Methods: 

Robotic  surgeons  (N=92)  participated  in  four  simulation  tasks  using  the  dV-Trainer 
(Mimic  Technologies,  Seattle,  WA),  a  simulator  for  the  da  Vinci  surgical  system  (Intuitive 
Surgical,  Inc.,  Sunnyvale,  CA).  Each  subject  performed  the  exercises  with  zero  latency 
and  then  was  randomly  assigned  a  fixed  latency  between  100-1000  ms  to  repeat  the 
exercises  (N=71  completed).  The  simulator  recorded  performance  data. 

Next,  four  hospital  campuses  within  25  miles  were  selected  to  measure  the  latency  in 
transferring  a  video  recording  of  a  robotic  surgical  procedure.  Data  packet  delivery 
times  were  measured  by  software  as  the  video  was  transmitted  between  all  four 
locations  via  the  hospital’s  existing  communication  network. 

Results: 

With  latency,  we  found  that  subjects  with  less  than  200ms  could  perform  procedures 
successfully.  Subjects  between  300-500  ms  had  variable  results.  Subjects  with  latency 
above  600ms  generally  could  not  complete  exercises  successfully.  Subject  experience 
levels  were  not  correlated  with  performance  when  latency  was  present. 


Actual  communication  latency  between  hospital  campuses  ranged  between  1-5  ms,  with 
one  outlier  at  62  ms.  Length  of  video  transmission  ranged  between  1 1  and  70  minutes. 

Conclusions: 

The  measured  latency  between  four  hospital  campuses  using  existing  networks  was  40 
times  lower  than  necessary  for  successful  surgery.  We  conclude  that  telesurgery  using 
existing  robotic  and  communication  equipment  is  possible  right  now  within  a  multi¬ 
campus  hospital  system. 
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Simulator  System  Comparison 


•Intuitive  Surgical  Inc 
•Skills  Simulator 
•VR  Environment 
•Self-contained  PC  plugs 
into  the  surgeon’s 
console 


•Mimic  Technologies  Inc 
•dV-Trainer 
•VR  Environment 
•Unique  hardware  and 
software  device 


* 


•Surgical  Simulation  Inc 
•RoSS  Simulator 
•VR  Environment 
•Unique  hardware  and 
software  device 


Florida  Hospital! 

Nicholson  Center; 


Comparative  analysis  of  the  functionality  of 
simulators  of  the  da  Vinci  surgical  robot 


Roger  Smith,  PhD,  Florida  Hospital  Nicholson  Center 
Mireille  Truong,  MD,  Florida  Hospital  Celebration  Health 
Manuela  Perez,  MD,  Florida  Hospital  Nicholson  Center 
&  IADI-INSERM  U947  Nancy-FRANCE 


I 


Robotic  Simulator  Feature  Comparison 


Features 


DVSS 


dV-Trainer 


RoSS 


System  Manufacturer 


Intuitive  Surgical  Inc. 


Mimic  Technologies  Inc. 


Simulated  Surgical  Systems  LLC 


Specifications 
(Simulator  only) 


Depth  7" 

Height  25" 

Width  23" 

120  or  240V  power 


Depth  36" 

Height  26" 

Width  44" 

120  or  240V  power 


Depth  44" 

Height  77" 

Width  45" 

120  or  240V  power 


Specifications 

(Complete  System  as  shown  in 
Figure  1) 


Depth  41" 

Height  65" 

Width  40" 

120  or  240V  power 


Depth  36" 

Height  59" 

Width  54" 

120  or  240V  power 


Depth  44" 

Height  77" 

Width  45" 

120  or  240V  power 


Visual  Resolution 


VGA  640  x  480 


VGA  640  x  480 


VGA  640  x  480 


Components 


Customized  computer  attached  to 
da  Vinci  surgical  console 


Standard  computer,  visual  system 
with  hand  controls,  foot  pedals. 


Single  integrated  custom 
simulation  device 


Support  Equipment 


da  Vinci  surgical  console,  custom 
data  cable 


Adjustable  table,  touch  screen 
monitor,  keyboard,  mouse, 
protective  cover,  custom  shipping 
container 


USB  adapter,  keyboard,  mouse 


Exercises 


35  simulation  exercises 


56  simulation  exercises 


52  simulation  exercises. 


Optional  Software 


PC-based  Simulation  management 


Mshare  curriculum  sharing  web 
site 


Video  and  Haptics-based 
Procedure  Exercises  (HoST) 


Scoring  Method 


Scaled  0-100%  with  passing 
thresholds  in  multiple  skill  areas 


Proficiency- based  point  system 
with  passing  thresholds  in 
multiple  skill  areas 


Point  system  with  passing 
thresholds  in  multiple  skill  areas 


Student  Data  Management 


Custom  control  application  for 
external  PC.  Export  via  USB 
memory  stick. 


Export  student  data  to  delimited 
data  file. 


Export  student  data  to  delimited 
data  file. 


Curriculum  Customization 


None 


Select  any  combination  of 
exercises.  Set  passing  thresholds 
and  conditions. 


Select  specifically  grouped 
exercises.  Set  passing  thresholds. 


Administrator  Functions 


Create  student  accounts  on 
external  PC.  Import  via  USB 
memory  stick. 


Create  student  accounts. 
Customize  curriculum. 


Create  student  accounts. 
Customize  curriculum. 


System  Setup 


None. 


Calibrate  controls. 


Calibrate  controls. 


System  Security 


Student  account  I D  and 
password. 


PC  password,  Administrator 
password,  Student  account  ID 
and  password. 


PC  password,  Administrator 
password,  Student  account  I D 
and  password. 


Simulator  Base  Price 


$85,000 


$95,000 


$107,000 


Support  Equipment  Price 


$502,000 


$9,100 


$0 


Total  Functional  Price 


$587,000 


$104,100 


$107,000 


Simulator 


ability  Ex 


Subject 
14  7 

-*-)  V  7  /  ••• 


Subject 
2,  5,  8, ... 


Simulator 


Subject 
3,  6,  9, ... 


Questionnaire  #1: 

1 .  The  hand  controllers  on  this  simulator  are 
effective  for  working  in  the  simulated 
environment. 

2.  The  3D  graphical  exercises  in  the  simulator 
are  effective  for  teaching  robotic  skills. 

3.  The  hand  controllers  are  well  synchronized 
with  the  3D  graphical  world  objects. 

4.  The  device  is  a  sufficiently  accurate 
representation  of  the  real  robotic  system. 

5.  The  scoring  system  effectively 
communicates  my  performance  on  the 
exercise. 

6.  The  scoring  system  effectively  guides  me  to 
improve  performance  on  the  simulator. 

7.  I  believe  this  device  would  be  an  effective 
tool  for  maintaining  robotic  surgical  skills 
when  a  surgeon  is  not  able  conduct  regular 
operations. 

8.  Researcher  intervention  was  required. 
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Simulator  Effectiveness  Experiment 


Simulator  Simulator  Simulator 


A  B  C 

Achieve 

Proficiency 


Maintain 

Proficiency 


Two 

Months 


Final 

Assessment 


Simulator  Effectiveness  Objectives 


Conceptual  Results 


Simulator 

Scores 


Robotic 

Scores 


Doctors  vs.  Gamers 


•Mimic  Technologies  Inc 
•dV-Trainer 
•VR  Environment 
•Custom  hardware  and 
software  device 


Which  Group 

Masters 

- > 

Simulated 

Surgical  Tasks 

Faster? 

18 


Simulator  Exercise 


2 


1 


Comparative  Performance 


Conceptual  Results 


Simulator 

Scores 


Joe  Students  Gamers 


Comparative  Performance 


Conceptual  Results 


120 


Surgical  Rehearsal  with  Simulator 


Pre-Test  Questionnaire 


I  Randomized  Assignment  I 


Robotic  Procedure 


N) 


Suture  Training  Modes 


Randomized  Sub  ects 


No  Prep 


Evaluation  of  Skill  Level 


Evaluation  of  Skill  Level 


Fundamentals  of  Robotic  Surgery 

Create  and  develop  a  validated  multi-specialty,  technical  skills 
competency  based  curriculum  for  surgeons  to  safely  and 
efficiently  perform  basic  robotic-assisted  surgery. 


1.  Outcomes  Measures  (Dec  12-13, 2011) 

2.  Curriculum  Outline  (April  29-30, 2012) 

2.5  Curriculum  Development  (Aug  17-18, 2012) 

3.  Validation  Criteria  (Nov  17-18, 2012) 

4.  Online  Course  Materials  (Oct-Dec  2013) 

5.  Psychomotor  Device  Development  (Oct-Dec  2013) 

6.  Validation  Studies  (2014) 

7.  Transition  to  Objective  Testing  Organization  (est.  2015) 
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FRS  Outcomes  Measures 


Pre-Operative 

Intra-Operative 

Post-Operative 

System  Settings 

Energy  Sources 

Transition  to  Bedside  Asst 

Ergonomic  Positioning 


Docking 


Robotic  Trocars 


OR  Set-up 


Situation  Awareness 


Closed  Loop  Comms 


Respond  to  System  Errors 


Camera  Control 


Clutching 


Instrument  Exchange 


Foreign  Body  Management 


Multi-arm  Control 


Eye-hand  Instrument  Coord 


Wrist  Articulation 
Atraumatic  Tissue  Handling 


Dissection  -  Fine  &  Blunt 
Cutting 

Needle  Driving 


Suture  Handling 


Knot  Tying 


Safety  of  Operative  Field 


Undocking 


See  separate  file 


y t  Psychomotor  Device  Evolution 


Prototype  Concept  Prototype  1  Prototype  2  Prototype  3 
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Virtual  Worlds  for  Robotic  Surgery 


HumanSim  Preview  for  iPad  is  available  on  iTunes 


Robotic  Surgery  Education  Evolution 


Coming  to  your  local  mall ... 
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FLORIDA  HOSPITAL 

Nicholson  Center 


Gamers  in  Surgical  Simulation: 

A  Comparison  of  Gamers,  Surgeons,  Medical 

Students,  and  Clinical  Staff 


Roger  Smith  &  Alyssa  Tanaka 

roger.smith@flhosp.org 

alyssa.tanaka@flhosp.org 


Gamers  vs.  Doctors  on  Simulators 


Orlando  Area  Collaboration 


UNlVlHSItYOf  ClNIHAl  I IOHI1M 

College  ^/'Medicine 


mmic 


Experiment 


Hypothesis  1:  Medical  students,  clinical  staff,  and  lay  persons  achieve 
equivalent  performance  on  the  dV-Trainer. 

Hypothesis  2:  VGEs  are  equivalent  in  simulation-based  skills  to 
experienced  surgeons. 

Hypothesis  3:  VGEs  scores  on  psychological  tests  of  perceptual  and 
cognitive  skills  are  equivalent  to  those  of  expert  surgeons. 

Subjects:  FH  Surgeons,  UCF  Medical  Students,  UCF  FIEA  &  Full  Sail 
Game  Programming  Students,  FH  Clinical  Staff,  FH  Laypersons 

Device:  dV-Trainer  Simulator  (Mimic  Technologies  Inc.) 

Tasks:  Questionnaires,  Perceptual  Tests,  Simulation  Exercises 


Experimental  Process 


Activity 

Estimated 

Time  to  Complete 

Consent 

5 

Pre-test  Questionnaire 

5 

Perceptual  Tests 

10 

dV-Trainer  Warmup 

3 

8  Trials  on  the  dV-Trainer 

40 

Post-Test  Questionnaire 

5 

Total  Time 

68  mins 

Perceptual  Tests 


CL 


Flanker  Compatibility 


CTioamii 


Enumeration  task 


Simula 

=  -Lx,g  t-h- 


Ring  &  Rail 


Suturing 


Comparative  Performance 


Conceptual 


Simulator 

Metrics 


Powering  the  Study 


Task 

Mean  Score 

Minimum  and 
Maximum  Diff 

Difference  detected  at 
80%  power  with 
N=125  fA/21 

1055 

14  (79) 

25 

371 

14  (907) 

192 

Task 

Mean  Time  to 
Complete 

Minimum  and 
Maximum  Diff 

Difference  detected  at 
80%  power  with 
N=125  (A/ 2) 

68 

3  (29) 

9 

682 

67 (543) 

120 

Medical  Student  Performance 


Medical  Student  Performance 


Medical  Student  Performance 


Ring  &  Rail:  Overall  Score 


1200 


KC1CI 


■GOO 


T  ria  I 


Suture  Sponge:  Overall  Score 


Ln 


1300 


1000 


MCK3 


GOO 


A  CKJ 


200 


3 


3 


Jt 


£ 


* 


H 


Trial 


H2:  Potentially 
Lay  Persons  are 
not  viable  subjects 


Value  of  Results 


H2:  Potentially  Clinical  Staff  can 
be  used  as  novice  subjects  in 
some  studies  of  robotic  surgery 


HI:  Potentially  VGEs  should  be 
excluded  from  novice  populations 
in  robotic  surgery  studies 


H3:  Potentially  the  perceptual  abilities  of  Expert  Surgeons  and  VGEs  are 
similar. 


The  Effectiveness  and 
Usability  of  Robotic 
Simulators 


Alyssa  Tanaka,  Mireille  Truong  MD, 
Khara  Simpson  MD,  Manuela  Perez 
MD,  Roger  Smith  PhD 


Florida  hospital 


Nicholson  Center 


*  Introduction 

*  Hypothesis 

*  Methods 

*  Results 

*  Conclusion 


k  HOSPTTAL 

son  Center 


Skills  Simulator 

dV-Trainer 

RoSS  Simulator 

•Intuitive  Surgical  Inc 
•VR  Environment 
•Self-contained  PC  plugs 
into  the  surgeon's  console 


•Mimic  Technologies  Inc 
•VR  Environment 
•Unique  hardware  and 
software  device 


•Surgical  Simulation  Inc 
•VR  Environment 
•Unique  hardware  and 
software  device 


Introduction 


•  Face  Validity:  The  realism  of  the  simulator. 

•  Content  Validity:  Judgment  of  the 
appropriateness  of  the  simulator  as  a  teaching 
modality. 

•  Concurrent:  The  extent  to  which  the  simulator 
correlates  with  the  "gold  standard." 

•  Construct  Validity:  Indicates  whether  the 
simulator  is  able  to  distinguish  an  experienced 
surgeon  from  an  inexperienced  surgeon. 

•  Predictive  Validity:  The  extent  to  which  the 
simulator  predicts  future  performance. 


McDougall,  E.  M.  (2007).  Validation  of  surgical  simulators.  Journal  of  Endourology,  21(3),  244-247. 


Florida  hospital 

Nicholson  Center 


Introduction 


Phase  1:  Systems  Capabilities  Evaluation 


Phase  2:  Subjective  Usability  Analysis 


Phase  3:  Objective  measurement  of  skill 

Acquisition 


Florida  hospttal 

Nicholson  Center 


Hypothesis 


H0:  There  is  no  difference  between  the 
content,  face  and  construct  validity  of  the 
three  simulators. 

Ha:  There  is  a  difference  between  the  content, 
face  and  construct  validity  of  the  three 
simulators. 


Florida  hospttal 

Nicholson  Center 


Novice:  0-19 
robotic  cases 

Intermediate: 

20-99  robotic 
cases 

Expert:  100+ 
robotic  cases 


Simulator 

A 


Simulator 
C 


Subject 
14  7 


Subject 
2,  5,  8, ... 


Subject 
3,  6,  9, ... 


n  Center 


Methods 


Pretest 


* 


Florida  hospttal 

Nicholson  Center 


Simulator  Demonstration 


[Video  clip  from  all  3  devices  was  embedded  here.] 


Florida  hospttal 

Nicholson  Center 


Results 


Novices:  n=  36  Intermediates:  n=30  Experts:  n=  34 


What  would  you  choose 
for  trainees? 


dv  trainer 
DVSS 

dv  trainer  and 
DVSS 


Overall,  which  did  you 
most  like? 

■  RoSS 


dv  trainer  and 
DVSS 


Which  was  physically 
most  comfortable? 


dv  trainer  and 
DVSS 


Which  made  you  feel 
stressed? 


■  RoSS 

■  dv  trainer 

■  DVSS 

■  dv  trainer  and 
RoSS 

None 


FLORIDA  HOSPITAL 

Nicholson  Center 


Results 


Florida  hospttal 

Nicholson  Center 


Conclusions 


Beginning  analysis 

—  Face  and  content  validity  from  questionnaires 

—  Construct  validity  from  simulation  scores 

Give  guidance  to  purchasers  on  the  simulator  most  preferred 

—  RoSS 

•  least  preferred  in  terms  of  usability 

•  Not  worth  the  cost 

—  dv-trainer 

•  Worth  the  cost 

—  Skills  simulator 

•  Most  preferred 

•  Worth  the  cost 

Future  Research:  objective  evaluation  of  skill  acquisition  and  retention 
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training  methods,  regardless  of  their  assigned  group.  Subjects  subsequently  performed  a  robotic 
cystotomy  repair  on  a  live  porcine  model,  followed  by  completion  of  a  post-procedure 
questionnaire. 

Results: 

Both  groups  (total  n=  125;  DG  n  =  64;  SG  n  =  61)  were  similar  in  age,  gender,  role,  and  total  number 
of  robotic  cases.  The  majority  of  subjects  had  prior  robotic  simulator  use,  DG>SG  (p  =  0.04).  Both 
groups  felt  that  neither  one  training  method  alone  was  sufficient  for  robotic  surgical  training,  both 
before  and  after  the  training  session.  The  didactic  group  (70%)  felt  more  prepared  than  their 
simulation  counterparts  (45%)  to  perform  a  robotic  procedure  after  their  training  session.  However, 
both  groups  would  select  simulation  for  training  if  they  could  only  chose  one  method  (SG  87%,  DG 
90%  before  training;  SG  93%,  DG  94%  after  training). 

Conclusions: 

Although  surgical  trainees  and  trainers  agree  that  simulation  should  be  included  in  robotic  surgical 
training,  a  multi-modality  approach  that  includes  both  robotic  simulation  and  didactic  teaching  is 
preferred.  The  effectiveness  of  the  combination  of  both  methods  on  surgical  training  and 
performance  need  to  be  further  explored. 
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GEARS  Study 


SLS  Abstract 

Validating  the  Efficacy  of  GEARS  through  the  Assessment  of  100  Videos 

Objective:  To  evaluate  the  use  of  the  GEARS  assessment  to  distinguish  varying  robotic  surgical 
experience.  To  identify  additional  variables  that  may  increase  the  reliability  and  accuracy  of  GEARS. 

Methods:  Videos  were  collected  for  104  medical  students  and  surgeons  performing  a  simple  cystotomy 
closure  on  an  animate  model.  Subjects  were  divided  into  three  groups  based  on  their  robotic 
experience.  Reviews  were  performed  by  three  surgeons  (inter-rater  reliability  0.95  cronbach's  alpha) 
using  GEARS,  errors  and  tasks  assessments.  Statistical  analysis  was  performed  to  determine  the 
validity  and  internal  consistency  of  GEARS  as  well  as  its  correlation  with  the  independently  developed 
errors  and  tasks  check  lists. 

Results:  There  were  differences  between  task  time,  GEARS,  error  and  task  metrics  across  experience 
groups  (p  =  0.01).  For  the  individual  categories  of  GEARS,  all  were  able  to  differentiate  between 
experience  groups.  Bimanual  dexterity  and  efficiency  were  the  best  at  differentiating  intermediates  (26- 
100  robotic  cases)  from  experts  (>100).  In  the  evaluation  of  errors,  missed  targets,  needle  drops,  tissue 
damage,  and  instrument  collisions  were  statistically  different  among  groups  and  instruments  out  of 
view,  needle  re-loading,  tissue  re-grasping  were  not.  Instrument  collision  and  needle  drops  showed  the 
greatest  differentiation  among  groups.  GEARS  had  internal  consistency  (cronbach's  alpha  0.88)  and 
construct  validity  (p=<0.001). 

Conclusions:  This  is  the  first  study  to  confirm  the  ability  of  GEARS  to  differentiate  robotic  surgical 
experience  at  all  levels  of  training.  While  errors  and  task  assessment  also  showed  differences  among 
experience  levels,  they  were  less  reliable  than  GEARS. 


INTRODUCTION: 

The  inception  of  robotic  surgery  has  changed  the  face  of  surgery  by  providing  an  additional  means  to 
completing  more  complex  procedures  in  a  min.  invasive  fashion.  While  there  are  obvious  parallels  with 
traditional  laparoscopic  surgery,  endowrist  manipulation  and  lack  of  haptic  feedback  represent 
significant  differences.  Due  to  its  unique  interface,  robotics  has  necessitated  device  specific  training  and 
evaluation.  To  this  aim,  several  robotic  surgical  simulators  have  been  developed  and  validated.  The 
Fundamentals  of  Robotic  Surgery  (FRS)  is  also  being  trialed  and  validated  to  offer  specialty  non-specific 
didactic  and  simulation  training,  including  a  high  stakes  test. 

For  objective  in-OR  and  video  assessment,  the  global  evaluative  assessment  of  robotic  skills  (GEARS)  is 
used  most  often.  GEARS,  similar  to  GOALS  (a  validated  laparoscopic  objective  assessment  tool),  is  a  six- 
item  objective  assessment  using  a  Likert  scale.  The  six  fundamental  skills  assessed  are:  depth 
perception,  bimanual  dexterity,  tissue  handling,  efficiency,  autonomy,  and  robotic  control.  The  first  five 


skills  are  the  same  as  GOALS.  The  five  point  likert  scale  has  definitions  associated  with  3  of  the  5  scores. 
GEARS  has  been  found  to  have  construct  validity,  and  good  intra  and  inter-rater  reliability  (Goh  et  al 
2012).  No  studies  have  been  done  evaluating  its  ability  to  differentiate  levels  of  expertise  and  which 
skills  sets/categories  best  correlate  to  proficiency/surgeon  expertise  level.  We  propose  to  evaluate  the 
above  as  well  as  to  test  additional  structured  objective  assessments  (error  based  and  task  based  scoring) 
to  better  categorize  GEARS. 

Our  hypothesis  is  that  collisions  and  tissue  damage  will  be  highly  associated  with  novice  performance 
and  instrument  out  of  view  will  be  a  less  discriminatory  factor  for  experts. 


BACKGROUND 

GOALS  was  created  from  the  concept  of  OSATS  which  includes  the  objective  assessment  of  skills  and  a 
task  assessment. 

METHODS 

Archived  videos  and  assessments  were  reviewed  for  100  surgeons  performing  a  simple  cystotomy 
closure.  Previous  assessements  included  the  GEARS  and  an  addendum  questionnaire  regarding  errors 
and  task  assessment.  The  errors  were  designed  to  correlate  with  each  of  the  GEARS  items.  (Look 
below).  Reviews  were  performed  by  three  independent  reviewers  whose  interrater  reliability  was  .95 
via  cronbach's  alpha.  The  trained  reviewers  were  all  familiar  with  robotic  surgery  and  included  two 
minimally  invasive  fellows  and  an  attending  surgeon.  Training  consisted  of  reviewing  the  GEARS, 
encouraging  full  use  of  the  scale,  and  a  focus  on  documentation  of  only  confirmed  errors.  Reviewers 
were  blinded  to  the  experience  level  of  the  surgeons.  Intra  questionnaire  correlation  was  performed 
within  the  GEARS  as  well  as  with  the  addendum  questionnaire  to  determine  internal  consistency.  An  ICC 
was  performed  for  the  total  GEARs  score  as  well  as  with  the  deletion  of  each  item.  A  similar  process  was 
performed  for  the  task  and  error  portion  of  the  questionnaire.  Finally  the  overall  scores  of  each  (GEARS, 
task,  error)  were  compared  to  see  if  there  was  a  relationship.  The  mean  and  median  scores  were 
calculated  for  each  based  on  experience  levels.  Compare  each  item  with  experience  level  to  determine 
which  item  most  correlated  with  the  three  experience  categories.  Scatterplot  graphs  were  performed 
comparing  the  number  of  robotic  surgeries  peformed  with  the  total  GEARS  scores,  errors,  and  tasks. 
Confirm  that  the  developed  errors  and  task  questions  correlated  with  the  initially  identified  items  in 
GEARS. 

GENERAL  ANALYSIS 

1.  Mean  and  median  score  for  each  experience  group  for  the  GEARS,  errors,  and  task  assessment. 
Determine  if  these  differences  are  statistically  different.  HYPOTHESIS:  there  will  not  be  a 
statistically  significant  difference  between  intermediates  and  experts  but  it  will  be  present 
between  novices  and  experts.  RESULTS -There  were  statistically  significant  differences  across 
all  experience  levels. 


2.  Identify  which  components  of  the  GEARS  or  errors  or  task  assessments  where  there  are 
statistically  significant  differences  -  item  by  item  by  analysis. 

3.  Looking  at  correlation  between  the  items  with  no  statistical  significance  to  determine  if 
associated  task  or  error  components  gave  better  differentiation  between  the  two  groups. 

SUB-ANALYSIS 

1.  Identify  relationships  between  the  associations  between  GEARS  items  and  the  errors  and  task 
(see  below)  HYPOTHESIS: 

2.  Come  up  with  a  total  score/equation  including  the  GEARS,  errors  (neg  scoring),  and 
tasks.. ..compare  that  to  the  different  experience  levels  and  see  if  there  is  a  statistically 
significant  difference  among  the  groups 

RESULTS 

CONCLUSION/DISCUSSION 

Considerations  -  We  did  not  fully  evaluate  the  benchmark  of  autonomy.  As  noted  above  subjects  were 

given  the  same  task  to  complete  without  assistance.  Reviewers  in  most  cases  gave 


LIT  REVIEW:  Google  scholar  search  term  GEARS 


GEARS:  Original  study 

Jan  2012  Goh  et  al  -  Journal  of  Urology 

-  Used  expert  consensus  to  identify  an  objective  assessment  tool  for  the  eval  of  robotic  surgeons  - 
GOALS  with  an  additional  variable  for  robotic  control 

-  Determined  to  have  good  internal  consistency  and  reliability.  Also  established  construct  validity  but 
interestingly  enough  could  not  differentiate  PGY-6  (avg  surgical  cases  30)  from  expert  level  surgeons 
(avg  surgical  cases  190) 

-  4  attg  surgeons,  25  trainees  (  PGY4-6) 

Teaching  surgical  skills  -  Reznick  NEJM  2006 

Reviews  changes  in  medical  education  and  considerations  for  the  future 

Traditional  halstedian  model  of  education  stems  from  education  through  volume  of  patients.  With 
current  advances  in  medicine  the  people  are  living  longer,  increasing  the  complexity  of  conditions. 
Certain  conditions  are  becoming  increasingly  more  rare  as  well.  We  also  have  to  contend  with  work  hour 
shortages.  To  improve/strengthen  training  and  exposure  -  we  are  turning  to  simulation  for  orientation 
and  to  improve  outcomes. 


Learning  Tools  and  simulation  in  Robotic  Surgery:  State  of  the  Art  -  Citation  Pubmed 

-Fitts  and  Posner's  3  stage  theory  of  motor  skill  acquisition  -  suggests  that  in  the  first  phase  the  learner 
is  more  focused  on  the  steps  of  the  material  and  typically  is  less  coordinated.  This  suggests  that  maybe 
this  stage  of  learning  should  be  performed  in  a  non-lethal  environment 

Perform  searches  on  FLS/OSATS/Goals/Include  RTN 

Conclusions  -  (Discussion)  Unable  to  differentiate  between  intermediate  and  expert  level  surgeons  with 
GEARS  alone.  Need  another  variable  or  term  for  surgical  skills. 

-Limitations  -  Did  not  fully  assess  autonomy  -  everyone  defaulted  to  a  5. 

Meeting  Notes  4/28/14 

Correlations  between  GEARS  and  errors 

1.  Depth  perception  -  Intstrumentation 

2.  Bimanual  dexterity  -  Needle  Handling  and 

3.  Tissue  Handling -Tissue  handling 

4.  Efficiency -Time 

5.  Autonomy  -  Nothing/didn't  fully  evaluate 

6.  Robotic  Skills  -  Instrumentation  and  needle  handling  and  tasks 
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ABSTRACT 

Background:  Simulation  was  first  introduced  into  robotic  surgical  training  with  the  goal  of 
teaching  surgeons  new  skills  and  assessing  a  surgeon’s  skill  acquisition  over  time.  A  number  of 
robotic  simulators  have  been  introduced  to  support  these  aims.  The  three  daVinci  surgical  robotic 
simulators  that  are  currently  available  for  training  include  the  daVinci  Skills  Simulator  (Intuitive 
Surgical),  the  dV-Trainer  (Mimic  Technologies)  and  the  RoSS  (Simulated  Surgical  Systems). 
Given  that  the  introduction  of  these  technologies  is  still  relatively  recent  and  the  field  of 
minimally  invasive  surgery  has  seen  rapid  advancement,  there  is  a  need  to  develop  effective 
training  curriculums  and  to  identify  simulation  devices  that  will  best  achieve  these  goals.  While 
there  have  been  multiple  validation  studies  evaluating  each  of  these  simulators,  to  our 
knowledge,  there  are  no  other  studies  that  compare  the  manufacturer  specifications  and  the 
usability  of  all  three  robotic  simulators. 

Purpose:  To  compare  the  value  and  usability  of  the  three  daVinci  robotic  surgical  system 
simulators  while  evaluating  for  face  and  content  validity. 

Methods:  The  first  phase  of  the  analysis  is  an  objective  review  and  comparison  of  the  system 
design  and  capabilities  of  each  of  the  simulators.  A  systematic  literature  review  was  conducted  to 
retrieve  information  on  the  training  exercise  modules,  scoring  systems,  costs,  educational  impact, 
and  validation  methods  of  each  of  the  three  simulators.  The  findings  from  the  literature  search 
were  then  submitted  to  the  manufacturers  of  each  system  for  the  review  of  the  accuracy  of  the 
information. 

The  second  phase  of  the  analysis  is  a  subjective  evaluation  of  the  usability  of  the  systems.  Data 
is  being  gathered  via  opinion-based  questionnaires  from  medical  students  and  surgeons  of 
various  expertise  levels.  The  participants  first  complete  a  demographics  questionnaire  detailing 
their  background  information  including  their  experience  with  minimally  invasive  surgery  and 
robotic  surgery.  Subjects  then  complete  one  exercise  on  each  of  the  simulators  in  a  randomly 
assigned  order  and  provide  feedback  describing  their  experience  immediately  following 
completion  of  each  exercise.  After  completing  all  exercises,  the  participants  complete  a  final 


questionnaire  providing  feedback  on  their  overall  experience  using  all  three  simulators.  Exercises 
selected  on  each  simulators  tested  similar  skills  and  were  of  same  difficulty  level. 

Results:  The  results  of  the  first  phase  of  the  study  detail  the  characteristics,  exercise  modules, 
scoring  systems,  costs,  validity,  advantages,  and  disadvantages  of  each  simulator  and  provide  a 
comprehensive  chart  comparison  of  the  three  simulators. 

For  the  second  phase  of  the  study,  we  will  perform  an  interim  analysis  of  subjects  completed  to- 
date.  This  analysis  will  be  presented  at  the  conference. 

Conclusion:  Each  of  the  simulators  possesses  specific  qualities  and  capabilities  that  could  be 
beneficial  to  a  variety  of  different  users  and  training  programs. 
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Title:  A  prospective  randomized  controlled  comparative  study  on  surgical  training  methods  and  impact  on  surgical 
performance:  virtual  reality  robotic  simulation  vs  didactic  lectures 
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Objective:  To  compare  two  learning  methods,  structured  didactic  lecture  versus  virtual  reality  surgical  simulation, 
and  evaluate  their  effects  on  surgical  performance. 


Design:  Prospective  randomized  trial 


Setting:  Community  teaching  hospital  and  robotic  gynecologic  surgery  courses 


Patients:  Medical  students  (n  =  6),  residents  (n  =  14),  fellows  (n  =  28),  and  attendings  (n  =  76)  were  enrolled 


Interventions:  Subjects,  based  on  surgical  experience  level,  were  randomly  assigned  to  either  the  Simulation 
Group  (SG),  which  rehearsed  on  the  dv-Trainer  robotic  simulator;  or  to  the  Didactic  Group  (DG),  which  received  a 
structured  lecture.  After  completing  a  written  post-preparation  test,  all  participants  performed  a  video-recorded 
cystotomy  repair  on  a  live  porcine  model.  Time  and  performances  were  compared  using  GEARS  and  independently 
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developed  error  and  task  metrics. 


Measurements  &  Main  Results:  Total  n=  125  (52  novices,  42  intermediates,  27  experts).  Both  groups  (DG 

n  =  64;  SG  n  =  61)  were  similar  in  age,  gender,  role,  and  total  number  of  robotic  cases.  The  majority  of  subjects  had 

prior  robotic  simulator  use,  DG>SG  (p  =  0.04).  Mean  cystotomy  repair  time  was  similar  in  both  groups 

(DG  =  224min;  SG  =  219min,  p=0.83).  The  overall  performances  between  SG  and  DG  were  not  significantly  different 

(p  value=  0.18  -  0.83)  but  when  controlled  for  experience  level,  SG  (vs  DG)  novices  had  more  errors  and  lower 

task  assessment  scores  (p  value  0.03  and  0.05).  No  differences  were  noted  between  learning  groups  amongst 

intermediates  and  experts. 

Conclusions:  Simulation  and  didactic  approaches  both  offer  certain  advantages  for  surgical  training.  Although 
differences  were  not  seen  between  the  two  training  methods,  many  factors  may  have  influenced  these  results  such 
as  simulator  training  type,  task  complexity  and  length;  realism  of  simulation  exercise  to  actual  procedure;  and 
differences  in  information  presented  with  each  method.  Both  cognitive  and  psychomotor  skills  are  required  for 
surgical  competence;  therefore  the  effect  of  the  combination  of  both  modalities  rather  than  a  single  modality  for 
surgical  training  should  be  further  explored. 
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